diff --git a/AGENTS.md b/AGENTS.md index e3816cc1..a789a8b9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -11,6 +11,7 @@ DICE (Domain-Integrated Context Engineering) is a proposition-first knowledge su | `dice-storage-autoconfigure` | Spring Boot auto-configuration that wires the right backend based on `embabel.dice.store.type`, schedules the decay tick, and provides auto-configuration for the multi-signal duplicate collector (properties prefix `embabel.dice.collector`) | | `dice-report` | Output projectors over propositions: rationale (why a fact is believed, with evidence), structured report, and surprising-link discovery | | `dice-ingestion` | Ingestion SPI (artifacts → chunks) with a content-hash dedup ledger so the same source isn't extracted twice | +| `dice-metamodel` | Schema versioning: `MetamodelVersion` content-hash stamps over the governed types of a `DataDictionary`, `GovernedTypeSelector`, the `DeclaredSchemaSource` opt-in, and the `MetamodelVersionStore` contract. Pure JVM, with no dependency on `dice` | | `dice-integration-tests` | Test-only: the cross-feature end-to-end canonical-flow harness | ## Build & test diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..cd5fa989 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,47 @@ +# Changelog + +Notable changes to DICE. Each entry states its compatibility impact on consumers +(anything tracking `0.2.0-SNAPSHOT`): **additive** (safe to +pick up), **behavioral** (same API, different runtime behavior — read the note), +or **breaking** (consumer change required; the entry links the migration notes +and the consumer PRs that deliver it). + +## Unreleased + +### Added + +- `dice-metamodel` module, first slice of schema versioning: `MetamodelVersion` + content-hash stamping with per-type governance selection, the declared-schema + opt-in seam, and the `MetamodelVersionStore` contract. Pure JVM. + **Compatibility: additive.** New module; no existing API touched. + +- Declared renames in `dice-metamodel`. **EXPERIMENTAL** (shape may change + before 1.0): `SchemaAliases`, `PropertySignature.aliases`, and + `MetamodelVersion.entityTypeAliases`. A declaration states the names a type or + property used to go by, so a later comparison pairs a rename instead of + reading it as a removal and an addition. + **Compatibility: additive.** `contentHash` is unchanged for any schema that + declares no aliases — the new hash blocks serialize only when non-empty, and + the pinned golden digest is asserted unchanged, including for a stamp rebuilt + through the public constructor. The new constructor parameters carry + `@JvmOverloads`, so the existing `PropertySignature(String, Kind, String, + Cardinality)` and `MetamodelVersion(String, List, Map, Map, List)` descriptors + survive. `SchemaAliases` reaches `MetamodelVersion.from` and + `DeclaredSchema.from` through separate overloads that take it as a required + parameter, which leaves the shipped one- and two-argument forms and their + Kotlin `from$default` synthetics byte-identical; widening those functions with + a third defaulted parameter would have replaced + `DeclaredSchema.Companion.from$default(Companion, DataDictionary, + GovernedTypeSelector, int, Object)` with a wider descriptor and broken any + caller already compiled against it. `MetamodelJavaCompatTest` calls every + static form. The changed Kotlin synthetic constructor, `copy`, `copy$default` + and `componentN` signatures on `PropertySignature` are the accepted boundary: + Kotlin callers recompile, and no consumer holds a compiled reference to them. + +- Schema attribution mechanism: per-proposition version is answered through the + run that produced the proposition (PRODUCED_BY_RUN). The run record carries + the declared schema's content hash, resolved by the extraction coordinator from + the host's DeclaredSchemaSource. The `DiceMetadataKeys.METAMODEL_VERSION` + metadata key is removed; lineage answers per-proposition attribution. + **Compatibility: breaking.** The key is no longer available; code holding it + must migrate to extraction-run queries. diff --git a/README.md b/README.md index d4a92ef2..fd9e296c 100644 --- a/README.md +++ b/README.md @@ -113,6 +113,8 @@ recover by reading a single class — see the design notes in [`docs/design/`](d - [Durable storage](docs/design/durable-storage.md) — backend selection, defense-in-depth dedup, two-phase save, materialised effective confidence, schema-as-beans, and the decay tick. - [Events](docs/design/events.md) — the domain-event model the store and pipeline emit. +- [Metamodel versioning](docs/design/metamodel-versioning.md) — content-hash schema stamps, per-type + governance, the declared-schema opt-in, and version history. ## Real-World Example: Impromptu diff --git a/dice-metamodel/pom.xml b/dice-metamodel/pom.xml new file mode 100644 index 00000000..67a95652 --- /dev/null +++ b/dice-metamodel/pom.xml @@ -0,0 +1,93 @@ + + + 4.0.0 + + com.embabel.dice + dice-parent + 0.2.0-SNAPSHOT + + dice-metamodel + jar + Dice Metamodel + Schema versioning for DICE knowledge graphs: content-hash stamping, the declared-schema contract, and the version store contract + + + + + com.embabel.agent + embabel-agent-api + provided + + + + + org.jetbrains + annotations + provided + + + + + org.springframework.boot + spring-boot-starter-test + test + + + + + + + + org.jetbrains.kotlin + kotlin-maven-plugin + + + -Xjvm-default=all + + + + + compile + compile + + compile + + + + src/main/java + src/main/kotlin + + + + + test-compile + test-compile + + test-compile + + + + src/test/java + src/test/kotlin + + + + + + + + + diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt new file mode 100644 index 00000000..469a6a46 --- /dev/null +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/DeclaredSchemaSource.kt @@ -0,0 +1,128 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +import com.embabel.agent.core.DataDictionary +import org.jetbrains.annotations.ApiStatus + +/** + * The schema as declared: the stamped [version] plus the bare relationship type names it allows. + * + * The bare names travel alongside the stamp rather than being recovered from it, because + * [MetamodelVersion.relationshipNames] holds rendered `From-[name]->To` descriptors and + * reverse-parsing one is ambiguous: these names come from free-text and LLM extraction, and can + * themselves contain a `-[...]->`-shaped substring. Whoever builds a declaration holds the + * un-rendered names before anything is stamped, so it passes them straight through. Comparing + * declared relationships against what a graph holds needs those bare names, and that comparison is + * the next slice. + * + * @property version The stamped declared schema. + * @property relationshipTypeNames The bare relationship type names [version] allows. + */ +@ApiStatus.Experimental +class DeclaredSchema( + val version: MetamodelVersion, + relationshipTypeNames: Set, +) { + + /** Copied into a JVM-immutable set so the declaration can't drift from its stamped [version]. */ + val relationshipTypeNames: Set = java.util.Set.copyOf(relationshipTypeNames) + + override fun equals(other: Any?): Boolean = + other is DeclaredSchema && + version == other.version && + relationshipTypeNames == other.relationshipTypeNames + + override fun hashCode(): Int = 31 * version.hashCode() + relationshipTypeNames.hashCode() + + override fun toString(): String = + "DeclaredSchema(version=$version, relationshipTypeNames=$relationshipTypeNames)" + + companion object { + + /** + * Declare the governed part of [dataDictionary]: stamp it and carry through the bare + * relationship names the same governed types declare. + * + * Use this rather than building the two halves separately. Picking a governed subset for + * the stamp while taking relationship names from the whole dictionary would declare + * relationships the stamp never covered, and the mismatch would only surface much later, + * as phantom disagreement in a drift check. + * + * @param dataDictionary The schema to declare. + * @param selector Which types are under governance. Defaults to all of them. + * @return The declaration. + */ + @JvmStatic + @JvmOverloads + fun from( + dataDictionary: DataDictionary, + selector: GovernedTypeSelector = GovernedTypeSelector.ALL, + ): DeclaredSchema = from(dataDictionary, selector, SchemaAliases.NONE) + + /** + * Declare the governed part of [dataDictionary], carrying the former names [aliases] + * declares for its types and properties. + * + * This is how a rename gets recorded as one. The stamp carries the former names, and a + * later comparison pairs the old name with the new one rather than reading the change as a + * type or property disappearing. + * + * [aliases] has no default, and the shorter form above stays a separate function. Adding a + * third defaulted parameter to it would replace its synthetic `from$default` descriptor + * with a wider one, which is a link error for any caller already compiled against it. + * + * @param dataDictionary The schema to declare. + * @param selector Which types are under governance. [GovernedTypeSelector.ALL] governs all + * of them. + * @param aliases Former names for the schema's types and properties. [SchemaAliases.NONE] + * declares none. Experimental: shape may change before 1.0. + * @return The declaration. + * @throws IllegalArgumentException when a declared type name appears in another type's + * alias set, or when aliases are declared for a property name the governed types hold + * more than one signature for. + */ + @JvmStatic + fun from( + dataDictionary: DataDictionary, + selector: GovernedTypeSelector, + aliases: SchemaAliases, + ): DeclaredSchema = DeclaredSchema( + version = MetamodelVersion.from(dataDictionary, selector, aliases), + relationshipTypeNames = MetamodelVersion.governedRelationshipTypeNames(dataDictionary, selector), + ) + } +} + +/** + * Supplies the schema an application has declared. + * + * This interface is the versioning opt-in: with no declared schema, nothing is stamped. The Spring + * wiring that arrives in a later slice activates only when a `DeclaredSchemaSource` bean is + * present, so an application that hasn't decided what it governs is left alone. + * + * A declared schema can come from anywhere. A consuming app implements this over whatever it + * already uses to define its types (a `DataDictionary`, a config file, a registry...) and wires it + * as a bean. There is no default implementation, because there is no default declared schema. + */ +@ApiStatus.Experimental +fun interface DeclaredSchemaSource { + + /** + * @return the current [DeclaredSchema]. + */ + fun declare(): DeclaredSchema +} diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/GovernedTypeSelector.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/GovernedTypeSelector.kt new file mode 100644 index 00000000..1c77ee48 --- /dev/null +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/GovernedTypeSelector.kt @@ -0,0 +1,59 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +import com.embabel.agent.core.DomainType +import org.jetbrains.annotations.ApiStatus + +/** + * Decides which domain types a [MetamodelVersion] stamp covers. + * + * A DICE domain is rarely all one thing. Part of it is closed-world: the types you have committed + * to, whose shape you want to notice changing. The rest is open-world: exploratory types that + * extraction proposes, that come and go, and that would otherwise break a version comparison every + * time an LLM invents one. Governance is therefore per type and opt-in, in the spirit of + * Hibernate's `@Version`. + * + * A selector is a predicate over types, so the usual form is a lambda over names you already hold: + * + * ```kotlin + * val governed = setOf("Person", "Company") + * val version = MetamodelVersion.from(dataDictionary, GovernedTypeSelector { it.name in governed }) + * ``` + * + * Selecting a subset changes which types the stamp covers, and leaves the encoding alone. Adding an + * ungoverned type to the dictionary leaves the content hash as it was, while touching a governed + * one changes it. + */ +@ApiStatus.Experimental +fun interface GovernedTypeSelector { + + /** + * @param type A domain type from the dictionary being stamped. + * @return `true` when this type is under version governance and belongs in the stamp. + */ + fun governs(type: DomainType): Boolean + + companion object { + + /** + * Governs every type in the dictionary. This is the whole-schema stamp, and what + * [MetamodelVersion.from] uses when no selector is given. + */ + @JvmField + val ALL: GovernedTypeSelector = GovernedTypeSelector { true } + } +} diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt new file mode 100644 index 00000000..aa9170eb --- /dev/null +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersion.kt @@ -0,0 +1,536 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +import com.embabel.agent.core.Cardinality +import com.embabel.agent.core.DataDictionary +import com.embabel.agent.core.DomainTypePropertyDefinition +import com.embabel.agent.core.NamedPropertyDefinition +import com.embabel.agent.core.PropertyDefinition +import org.jetbrains.annotations.ApiStatus +import java.security.MessageDigest +import java.util.Objects + +/** + * What a property looks like structurally: its name, whether it holds a plain value or points at + * another type, what that value or target is, and how many of them there can be. + * + * The stamp holds signatures rather than bare names. Turning `age` from a string into an integer, + * or a single `worksAt` into a list of them, changes what the graph can hold; with only names in + * the stamp, both would leave [MetamodelVersion.contentHash] untouched. + * + * Descriptions and semantic metadata are left out. They steer extraction, but they don't change the + * shape of what gets stored, so re-wording a description doesn't count as a schema change. + * + * @property name The property name. + * @property kind Whether the property holds a value or points at another domain type. + * @property type The value type (`string`, `integer`, ...), or the target type's name for a + * reference. Empty when [kind] is [Kind.UNKNOWN]. + * @property cardinality How many values the property holds: optional, one, a list, or a set. + * @property aliases Names this property used to go by, so a rename pairs up instead of reading as a + * removal and an addition. Declared through [SchemaAliases] and empty unless someone declared + * them. Part of the signature, so an alias-only edit moves [MetamodelVersion.contentHash]. + * Held as handed in, including a mutable set: a `data class` component can't be normalised on the + * way through, since a constructor `val` takes no initialiser and the generated `copy` calls that + * same constructor. [MetamodelVersion] therefore copies the set into an immutable one when it + * takes a signature, and a signature that never reaches a stamp keeps whatever set it was built + * with. Experimental: shape may change before 1.0. + */ +@ApiStatus.Experimental +data class PropertySignature @JvmOverloads constructor( + val name: String, + val kind: Kind, + val type: String, + val cardinality: Cardinality, + val aliases: Set = emptySet(), +) : Comparable { + + /** Whether a property holds a value of its own or points at another type in the schema. */ + enum class Kind { + /** A plain value: string, integer, and so on. */ + VALUE, + + /** A reference to another domain type, which shows up as a relationship in the graph. */ + REFERENCE, + + /** + * A property kind the agent platform has added that this module doesn't recognise. The + * name and cardinality still count towards the hash; the type is recorded as empty. + */ + UNKNOWN, + } + + override fun compareTo(other: PropertySignature): Int = ORDER.compare(this, other) + + companion object { + + /** + * Total order used to canonicalise a property set before hashing. Aliases come last, so a + * set holding two signatures that differ only in their aliases still sorts the same way + * every time. + */ + private val ORDER = compareBy({ it.name }, { it.kind }, { it.type }, { it.cardinality }) + .thenComparator { left, right -> compareAliases(left.aliases, right.aliases) } + + /** Compare two alias sets as sorted lists: element by element, then by size. */ + private fun compareAliases(left: Set, right: Set): Int { + left.sorted().zip(right.sorted()).forEach { (leftAlias, rightAlias) -> + val comparison = leftAlias.compareTo(rightAlias) + if (comparison != 0) return comparison + } + return left.size.compareTo(right.size) + } + + /** + * Read the structural signature off a property definition. + * + * @param property The property definition to summarise. + * @return Its structural signature. + */ + @JvmStatic + fun of(property: PropertyDefinition): PropertySignature = when (property) { + is DomainTypePropertyDefinition -> + PropertySignature(property.name, Kind.REFERENCE, property.type.name, property.cardinality) + + is NamedPropertyDefinition -> + PropertySignature(property.name, Kind.VALUE, property.type, property.cardinality) + + else -> + PropertySignature(property.name, Kind.UNKNOWN, "", property.cardinality) + } + } +} + +/** + * An immutable stamp that captures the identity and structural content of the governed part of a + * [DataDictionary] at a point in time. + * + * Two [MetamodelVersion] instances with the same [contentHash] represent semantically equivalent + * schemas. The extraction run records the version it ran under by carrying this content hash, and a + * proposition's version is answered through its run lineage. + * + * The hash is computed here from the structural fields, and a caller cannot supply it. It is the + * store's natural key and what [hasSameContentAs] compares, so if it were an ordinary constructor + * argument, two structurally different schemas could claim the same hash, compare as equal, and + * overwrite each other in storage. + * + * Everything handed to the constructor is copied into JVM-immutable collections and + * canonicalised: type and relationship lists come out sorted and free of duplicates. A stamp whose + * collections could still be changed from the outside, or that listed the same relationship twice + * because two same-named types declared it, would disagree with its own precomputed hash. This is + * a plain class rather than a `data class` for the same reason: a generated `copy()` would hand its + * arguments straight to the fields and skip all of that. + * + * @property schemaName The [DataDictionary.name] at the time the stamp was taken. + * @property entityTypeNames Governed entity type names, sorted and deduplicated. + * @property entityTypeLabels Full label set per type (including inherited labels), keyed by type + * name. Captured so label-only changes are detectable and reflected in [contentHash]. + * @property entityTypeProperties Full property signature set per type (including inherited + * properties), keyed by type name. Signatures rather than bare names, so a property changing type + * or cardinality moves [contentHash]. + * @property relationshipNames Rendered `From-[name]->To` descriptors for the relationships the + * governed types declare, sorted and deduplicated. + * @property entityTypeAliases Names each entity type used to go by, keyed by its current name, so a + * type rename pairs up instead of reading as a type vanishing and another appearing. Declared + * through [SchemaAliases] and empty unless someone declared them. Hashed, so an alias-only edit + * moves [contentHash]. Experimental: shape may change before 1.0. + * @property contentHash SHA-256 hex digest of the schema's entity types, label sets, property + * signatures, type aliases, and allowed relationships. The schema name is excluded so that two + * structurally identical schemas are equal regardless of how they are named. Stable across JVM + * restarts. Any structural change produces a different hash, including a property's type changing + * on a type whose name is unchanged. + */ +@ApiStatus.Experimental +class MetamodelVersion @JvmOverloads constructor( + schemaName: String, + entityTypeNames: List, + entityTypeLabels: Map>, + entityTypeProperties: Map>, + relationshipNames: List, + entityTypeAliases: Map> = emptyMap(), +) { + + val schemaName: String = schemaName + + val entityTypeNames: List = immutableCopy(entityTypeNames.distinct().sorted()) + + val entityTypeLabels: Map> = immutableCopy(entityTypeLabels) + + val entityTypeProperties: Map> = + immutableCopy(entityTypeProperties.mapValues { (_, signatures) -> signatures.map(::withImmutableAliases).toSet() }) + + val relationshipNames: List = immutableCopy(relationshipNames.distinct().sorted()) + + val entityTypeAliases: Map> = immutableCopy(entityTypeAliases) + + init { + // Only types named in entityTypeNames are walked when hashing, so a map entry keyed by + // anything else never reaches contentHash, and two stamps holding different labels or + // properties could share a hash and the store's natural key with it. `this.` is required + // here: the constructor parameters are still in scope and would shadow the copied fields. + val known = this.entityTypeNames.toSet() + val strayLabelKeys = this.entityTypeLabels.keys - known + require(strayLabelKeys.isEmpty()) { + "entityTypeLabels is keyed by types missing from entityTypeNames: ${strayLabelKeys.sorted()}. " + + "Labels for a type that isn't listed never reach contentHash." + } + val strayPropertyKeys = this.entityTypeProperties.keys - known + require(strayPropertyKeys.isEmpty()) { + "entityTypeProperties is keyed by types missing from entityTypeNames: ${strayPropertyKeys.sorted()}. " + + "Properties for a type that isn't listed never reach contentHash." + } + val strayAliasKeys = this.entityTypeAliases.keys - known + require(strayAliasKeys.isEmpty()) { + "entityTypeAliases is keyed by types missing from entityTypeNames: ${strayAliasKeys.sorted()}. " + + "Aliases for a type that isn't listed never reach contentHash." + } + + // An entry mapping a type to no former names at all hashes differently from having no entry, + // while saying the same thing, so two stamps of one schema could land on different keys. + val emptyAliasKeys = this.entityTypeAliases.filterValues { it.isEmpty() }.keys + require(emptyAliasKeys.isEmpty()) { + "entityTypeAliases holds empty alias sets for: ${emptyAliasKeys.sorted()}. " + + "Drop the entry instead; it means the same thing and hashes the same as the types around it." + } + + requireDeclarable(known, this.entityTypeAliases, this.entityTypeProperties) + } + + val contentHash: String = fingerprint() + + /** + * Returns `true` when this version and [other] have the same structural content (entity types, + * label sets, property signatures, and relationships), regardless of schema name. Compares + * [contentHash]. + * + * [equals] also compares [schemaName]; this does not. Use this to ask whether two schemas have + * the same shape, and [equals] to ask whether two stamps are the same stamp. + */ + fun hasSameContentAs(other: MetamodelVersion): Boolean = contentHash == other.contentHash + + /** + * A deterministic encoding of the structural fields, hashed with SHA-256 and rendered as + * lowercase hex. + * + * This is a persisted format. The digest is the store's natural key, so changing the encoding + * orphans everything already saved against it. `MetamodelVersionTest` pins the digest of a + * fixed schema with a literal assertion; changing the encoding means changing that literal + * deliberately and planning a migration. + */ + private fun fingerprint(): String { + // Every name, label and property component is length-prefixed, and each set is preceded by + // its size. These names come from free-text and LLM extraction and routinely contain ';', + // '[', '=' and spaces, so a delimiter-joined encoding could make ["a;b"] and ["a", "b"] + // hash identically and hide a lossy schema change. + // The schema name is excluded, so two structurally identical schemas produce the same hash + // even when named differently (e.g. dev vs prod environments). + // Alias blocks are written only when there is something in them, so a schema that declares + // no former names renders exactly the bytes this encoding produced before aliases existed + // and keeps every hash already recorded against it. Each block is `:|` followed + // by length-prefixed entries in sorted order, which is what the rest of the encoding does. + val hashInput = buildString { + append("types:").append(entityTypeNames.size).append('|') + entityTypeNames.forEach { name -> + appendSized(name) + appendAliasBlock("typealiases", entityTypeAliases[name].orEmpty()) + val labels = entityTypeLabels[name].orEmpty().sorted() + append("labels:").append(labels.size).append('|') + labels.forEach { appendSized(it) } + val properties = entityTypeProperties[name].orEmpty().sorted() + append("props:").append(properties.size).append('|') + properties.forEach { property -> + appendSized(property.name) + appendSized(property.kind.name) + appendSized(property.type) + appendSized(property.cardinality.name) + appendAliasBlock("aliases", property.aliases) + } + } + append("rels:").append(relationshipNames.size).append('|') + relationshipNames.forEach { appendSized(it) } + } + + val digest = MessageDigest.getInstance("SHA-256") + val hashBytes = digest.digest(hashInput.toByteArray(Charsets.UTF_8)) + return hashBytes.joinToString("") { "%02x".format(it) } + } + + /** + * Structural equality: same schema name, types, labels, property signatures, relationships, and + * aliases. + * + * This compares [schemaName]; [hasSameContentAs] and [contentHash] leave it out. Two stamps of + * identically shaped schemas under different names are therefore unequal here and equal there, + * and a `Set` keys the way the store does, on schema and content both. + */ + override fun equals(other: Any?): Boolean { + if (this === other) return true + if (other !is MetamodelVersion) return false + return schemaName == other.schemaName && + entityTypeNames == other.entityTypeNames && + entityTypeLabels == other.entityTypeLabels && + entityTypeProperties == other.entityTypeProperties && + relationshipNames == other.relationshipNames && + entityTypeAliases == other.entityTypeAliases + } + + override fun hashCode(): Int = Objects.hash( + schemaName, + entityTypeNames, + entityTypeLabels, + entityTypeProperties, + relationshipNames, + entityTypeAliases, + ) + + override fun toString(): String = + "MetamodelVersion(schemaName=$schemaName, contentHash=$contentHash, " + + "entityTypeNames=$entityTypeNames, entityTypeLabels=$entityTypeLabels, " + + "entityTypeProperties=$entityTypeProperties, relationshipNames=$relationshipNames, " + + "entityTypeAliases=$entityTypeAliases)" + + companion object { + + /** Append [token] length-prefixed (`:`) so concatenation can't be ambiguous. */ + private fun StringBuilder.appendSized(token: String) { + append(token.length).append(':').append(token) + } + + /** + * Append `:|` and the sorted, length-prefixed [aliases], writing nothing at all + * when there are none. + */ + private fun StringBuilder.appendAliasBlock(tag: String, aliases: Set) { + if (aliases.isEmpty()) return + append(tag).append(':').append(aliases.size).append('|') + aliases.sorted().forEach { appendSized(it) } + } + + /** Copy [values] into a JVM-immutable list, which a Java caller can't mutate via a getter. */ + private fun immutableCopy(values: List): List = java.util.List.copyOf(values) + + /** Copy a map of sets so both the map and every set inside it are JVM-immutable. */ + private fun immutableCopy(values: Map>): Map> = + java.util.Map.copyOf(values.mapValues { (_, set) -> java.util.Set.copyOf(set) }) + + /** + * Re-wrap a signature's alias set as a JVM-immutable one. The signature is a data class, so + * its alias set arrives however the caller built it; a caller who kept a mutable set and + * added to it afterwards would leave the stamp disagreeing with its own precomputed hash. + * + * The copy is unconditional. An empty set is the dangerous case, not the safe one: a caller + * holding an empty `mutableSetOf` can add to it after the stamp is built, which moves the + * signature's own `hashCode` while it sits in a hash-based set and leaves it unfindable in + * the collection that contains it. `Set.copyOf` of an empty set is a shared singleton, so + * skipping it buys nothing. + */ + private fun withImmutableAliases(signature: PropertySignature): PropertySignature = + signature.copy(aliases = java.util.Set.copyOf(signature.aliases)) + + /** + * The refusals a declaration has to pass, run as one so the constructor and [from] can't + * drift on which checks apply. [from] runs it first, so a declaration that can't be stamped + * fails at the call that stamped it; the constructor runs it again, so a stamp built by + * hand meets the same guard. + */ + private fun requireDeclarable( + declaredTypeNames: Set, + entityTypeAliases: Map>, + entityTypeProperties: Map>, + ) { + requireNoTypeAliasReuse(declaredTypeNames, entityTypeAliases) + requireNoAliasesOnDuplicateNames(entityTypeProperties) + } + + /** + * Reject a declared type name showing up in another type's alias set. + * + * One name would then belong to two types at once: the live type that carries it, and the + * renamed type that still claims it as a former name. Nothing downstream can tell which of + * the two a piece of data under that name belongs to. Reusing a retired name means first + * deleting the alias that still claims it. + */ + private fun requireNoTypeAliasReuse( + declaredTypeNames: Set, + entityTypeAliases: Map>, + ) { + entityTypeAliases.forEach { (typeName, aliases) -> + val reused = (aliases - typeName).filter { it in declaredTypeNames }.sorted() + require(reused.isEmpty()) { + "Entity type '$typeName' declares alias(es) $reused, and those name types the " + + "schema still declares. Retire the alias before reusing the name." + } + } + } + + /** + * Reject aliases on a property name a type holds more than one signature for. + * + * Two same-named domain types can each declare `age` with a different shape, and the union + * keeps both signatures. A comparison has no way to say which of the two an old name refers + * to, so the declaration is refused and the name compares as a removal plus an addition. + */ + private fun requireNoAliasesOnDuplicateNames( + entityTypeProperties: Map>, + ) { + entityTypeProperties.forEach { (typeName, signatures) -> + signatures + .groupBy { it.name } + .filterValues { withName -> withName.size > 1 } + .forEach { (propertyName, withName) -> + val declared = withName.flatMap { it.aliases }.distinct().sorted() + require(declared.isEmpty()) { + "Property '$propertyName' on entity type '$typeName' has " + + "${withName.size} signatures and declares alias(es) $declared. " + + "Retire the alias: a name with more than one signature can't say " + + "which one an old name meant." + } + } + } + } + + /** + * Create a [MetamodelVersion] stamp covering every type in [dataDictionary], which is the + * right stamp for a domain that is closed-world throughout. + * + * @param dataDictionary The schema to stamp. + * @return An immutable version stamp. + */ + @JvmStatic + fun from(dataDictionary: DataDictionary): MetamodelVersion = + from(dataDictionary, GovernedTypeSelector.ALL, SchemaAliases.NONE) + + /** + * Create a [MetamodelVersion] stamp covering only the types [selector] governs. + * + * DICE domains are usually part closed-world and part open-world. Some types are committed + * to, and their shape changing is worth noticing; the rest are exploratory, proposed by + * extraction, and churn by design. Stamping the whole dictionary makes that churn look like + * a schema change: every new exploratory type produces a new content hash, and the version + * history fills with entries nobody chose. Governance is therefore per type and opt-in, the + * way Hibernate's `@Version` is per entity. + * + * Adding, removing or reshaping an ungoverned type leaves [contentHash] untouched; doing + * the same to a governed one changes it. Relationships follow the type that declares them. + * A governed type's outgoing relationship is part of that type's declared shape and stays + * in the stamp even when it points at an ungoverned type, while a relationship declared + * *by* an ungoverned type is left out entirely. + * + * @param dataDictionary The schema to stamp. + * @param selector Which of its types are under governance. [GovernedTypeSelector.ALL] + * reproduces [from] exactly. + * @return An immutable version stamp covering the governed subset. + */ + @JvmStatic + fun from(dataDictionary: DataDictionary, selector: GovernedTypeSelector): MetamodelVersion = + from(dataDictionary, selector, SchemaAliases.NONE) + + /** + * Create a [MetamodelVersion] stamp covering the types [selector] governs, carrying the + * former names [aliases] declares for them. + * + * The upstream `PropertyDefinition` has nowhere to hold a former name, so aliases are + * declared alongside the dictionary and applied while the stamp is built: each signature + * picks up the former names declared for its property, and the type-level map is carried + * through for the governed types. Aliases declared for a type the selector doesn't govern + * are dropped, the same way everything else about an ungoverned type is. + * + * [aliases] has no default, and the two shorter overloads are separate functions rather + * than defaulted parameters on this one. A default here would fold all three into one + * function whose synthetic `from$default` descriptor replaced the shipped one, which is a + * link error for any caller already compiled against it. + * + * @param dataDictionary The schema to stamp. + * @param selector Which of its types are under governance. [GovernedTypeSelector.ALL] + * governs all of them. + * @param aliases Former names for the schema's types and properties. [SchemaAliases.NONE] + * declares none. Experimental: shape may change before 1.0. + * @return An immutable version stamp covering the governed subset. + * @throws IllegalArgumentException when a declared type name appears in another type's + * alias set, or when aliases are declared for a property name the governed types hold + * more than one signature for. + */ + @JvmStatic + fun from( + dataDictionary: DataDictionary, + selector: GovernedTypeSelector, + aliases: SchemaAliases, + ): MetamodelVersion { + val governedTypes = dataDictionary.domainTypes.filter { selector.governs(it) } + + // A DataDictionary can legally hold two domain types that share a name but differ in + // shape (DynamicType is a data class, so same-named instances with different labels are + // not equal and both survive a set). Labels and properties are unioned per name, off + // the one grouping. Keeping only the last would drop a label or property from the + // fingerprint, and removing it later wouldn't change the hash. + val typesByName = governedTypes.groupBy { it.name } + + val entityTypeLabels = typesByName.mapValues { (_, types) -> types.flatMap { it.labels }.toSet() } + + // Decorating inside this loop is the only place it can happen: the stamp is immutable + // and hashes at construction. Every signature sharing a property name picks up the same + // declared alias set, so decoration can neither create nor collapse a duplicate, and + // the constructor's duplicate-name guard sees exactly the duplicates the union holds. + val entityTypeProperties = typesByName.mapValues { (typeName, types) -> + types.flatMap { type -> + type.properties.map { property -> + val signature = PropertySignature.of(property) + val declared = aliases.propertyAliasesFor(typeName, signature.name) + if (declared.isEmpty()) signature else signature.copy(aliases = declared) + } + }.toSet() + } + + // Splitting one type into two same-named declarations, or merging two back into one, + // can render the same relationship descriptor twice. It is the same schema either way, + // so it has to be the same hash; the constructor sorts and deduplicates both lists. + val relationshipNames = dataDictionary.allowedRelationships() + .filter { selector.governs(it.from) } + .map { rel -> "${rel.from.name}-[${rel.name}]->${rel.to.name}" } + + val governedNames = typesByName.keys + val entityTypeAliases = aliases.typeAliases.filterKeys { it in governedNames } + + // Runs here as well as in the constructor so a declaration that can't be stamped fails + // at the call that stamped it, naming the alias to retire. + requireDeclarable(governedNames, entityTypeAliases, entityTypeProperties) + + return MetamodelVersion( + schemaName = dataDictionary.name, + entityTypeNames = governedNames.toList(), + entityTypeLabels = entityTypeLabels, + entityTypeProperties = entityTypeProperties, + relationshipNames = relationshipNames, + entityTypeAliases = entityTypeAliases, + ) + } + + /** + * The bare relationship type names the governed types declare: the same relationships + * [from] renders into [relationshipNames], un-rendered. + * + * Kept here so the governance rule (a relationship belongs to the type that declares it) + * lives in one place, and a [DeclaredSchema] can't drift from the stamp beside it. + */ + internal fun governedRelationshipTypeNames( + dataDictionary: DataDictionary, + selector: GovernedTypeSelector, + ): Set = dataDictionary.allowedRelationships() + .filter { selector.governs(it.from) } + .map { it.name } + .toSet() + } +} diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt new file mode 100644 index 00000000..62e06587 --- /dev/null +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelVersionStore.kt @@ -0,0 +1,94 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +import org.jetbrains.annotations.ApiStatus + +/** + * Durable store for metamodel version stamps. Keeping every stamp a schema has ever had is what + * later lets you say when a shape changed and what knowledge was extracted under which version. + * Implementations must preserve saved stamps exactly as recorded, including set contents and the + * ordering guarantees below. + * + * **What "save" means here.** [saveVersion] is an upsert on the natural key `(schemaName, + * contentHash)`. Saving a stamp whose key already exists matches the existing record and + * overwrites its non-key content, so no second record appears. That is idempotence in practice, + * because the key carries the content: the hash is derived from exactly the fields a re-save would + * overwrite, so anything landing on an existing key has identical content by construction. Nothing + * is deleted, and records with different keys always coexist, so history accumulates. + * Implementations are not expected to reject a re-save. + * + * **Why the schema name is in the key.** [MetamodelVersion.contentHash] excludes the schema name + * on purpose, so two schemas with the same shape share a hash: a schema and its staging copy, or a + * schema forked under a new name. History is per schema, which is what [latestVersion] and + * [versionHistory] answer, so the same content has to be a separate record under each name. + * Keyed on the hash alone, one schema adopting a shape another had earlier would land on the other + * schema's record and pull that schema's history into its own. The name in the key is what keeps + * two schemas' histories from bleeding into each other. + * + * This contract covers stamping and recall. Comparing a declaration against a live graph is a + * separate concern with its own store contract. + */ +@ApiStatus.Experimental +interface MetamodelVersionStore { + + /** + * Save a version stamp, keyed on `(schemaName, contentHash)`. Saving the same version twice + * leaves one stored version. + * + * @param version The version to save. + */ + fun saveVersion(version: MetamodelVersion) + + /** + * Return the most recently saved version for the given schema name, or `null` if no versions + * have been recorded. + * + * "Most recent" means logical write order. Re-saving an existing version does not make it the + * latest again, because the write matched a record that was already there. + * + * @param schemaName The schema to look up. + * @return The latest [MetamodelVersion], or `null` if unknown. + */ + fun latestVersion(schemaName: String): MetamodelVersion? + + /** + * Return all saved versions for the given schema, newest first. + * + * @param schemaName The schema to look up. + * @return All [MetamodelVersion]s for the schema, newest first. Empty if there are none. + */ + fun versionHistory(schemaName: String): List + + /** + * Look up one exact stamp by its natural key. + * + * Resolves a recorded hash, such as the one a proposition carries as the version it was + * extracted under, back into the schema shape it stood for. + * + * The default scans [versionHistory], which is correct for any implementation and reads the + * whole history to answer a keyed question. That is fine for the in-memory reference and for + * a test double. A durable backend should override it with a keyed lookup, since a long-lived + * schema's history only grows and this default grows with it; the Drivine store does, with a + * `MATCH` on the natural key. + * + * @param schemaName The schema the version belongs to. + * @param contentHash The [MetamodelVersion.contentHash] to find. + * @return The matching version, or `null` if this schema has no version with that hash. + */ + fun findVersion(schemaName: String, contentHash: String): MetamodelVersion? = + versionHistory(schemaName).firstOrNull { it.contentHash == contentHash } +} diff --git a/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/SchemaAliases.kt b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/SchemaAliases.kt new file mode 100644 index 00000000..28aea42c --- /dev/null +++ b/dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/SchemaAliases.kt @@ -0,0 +1,92 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +import org.jetbrains.annotations.ApiStatus + +/** + * The former names a schema's types and properties have gone by, declared alongside the schema + * itself. + * + * A rename with no alias behind it looks like a removal and an addition, which reads as loss. An + * alias tells the comparison that the two names are the same thing, so the rename pairs up. + * Iceberg and Delta get this from stable field ids assigned when a column is created; DICE + * extracts its schema from LLM output and has no id to assign, so the declaration carries the old + * name instead. + * + * Names are exact and case-sensitive. LLM extraction drifts on case, and treating `worksAt` and + * `worksat` as the same name here would quietly pair two properties an operator never said were + * the same one. + * + * Aliases accumulate. A type renamed `A` to `B` to `C` declares `{A, B}`, so a comparison across + * non-adjacent stamps still pairs. Retiring a name means deleting it from the declaration. + * + * Empty alias sets are dropped on the way in: declaring no former names for a type means the same + * thing as saying nothing about it, and an empty entry would otherwise change the content hash + * while meaning nothing. + * + * Experimental: shape may change before 1.0. + * + * @property typeAliases Current entity type name to the names it used to have. + * @property propertyAliases Entity type name, then current property name, to the names that + * property used to have. + */ +@ApiStatus.Experimental +class SchemaAliases @JvmOverloads constructor( + typeAliases: Map> = emptyMap(), + propertyAliases: Map>> = emptyMap(), +) { + + val typeAliases: Map> = copyDroppingEmpty(typeAliases) + + val propertyAliases: Map>> = java.util.Map.copyOf( + propertyAliases + .mapValues { (_, byProperty) -> copyDroppingEmpty(byProperty) } + .filterValues { it.isNotEmpty() }, + ) + + /** The former names declared for [propertyName] on [typeName], empty when none are. */ + fun propertyAliasesFor(typeName: String, propertyName: String): Set = + propertyAliases[typeName]?.get(propertyName).orEmpty() + + override fun equals(other: Any?): Boolean = + other is SchemaAliases && + typeAliases == other.typeAliases && + propertyAliases == other.propertyAliases + + override fun hashCode(): Int = 31 * typeAliases.hashCode() + propertyAliases.hashCode() + + override fun toString(): String = + "SchemaAliases(typeAliases=$typeAliases, propertyAliases=$propertyAliases)" + + companion object { + + /** No former names declared anywhere. What the stamping entry points use by default. */ + @JvmField + val NONE: SchemaAliases = SchemaAliases() + + /** + * Copy into a JVM-immutable map of JVM-immutable sets, leaving out keys whose alias set is + * empty. + */ + private fun copyDroppingEmpty(values: Map>): Map> = + java.util.Map.copyOf( + values + .filterValues { it.isNotEmpty() } + .mapValues { (_, aliases) -> java.util.Set.copyOf(aliases) }, + ) + } +} diff --git a/dice-metamodel/src/test/java/com/embabel/dice/metamodel/MetamodelJavaCompatTest.java b/dice-metamodel/src/test/java/com/embabel/dice/metamodel/MetamodelJavaCompatTest.java new file mode 100644 index 00000000..a2ac7a2d --- /dev/null +++ b/dice-metamodel/src/test/java/com/embabel/dice/metamodel/MetamodelJavaCompatTest.java @@ -0,0 +1,204 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel; + +import com.embabel.agent.core.Cardinality; +import com.embabel.agent.core.DataDictionary; +import com.embabel.agent.core.DomainType; +import com.embabel.agent.core.DynamicType; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.List; +import java.util.Map; +import java.util.Set; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Calls every entry point the way a Java consumer compiled against the previous release does. + *

+ * Aliases were added as trailing parameters with defaults, so a Kotlin caller sees no change. A + * Java caller sees whatever descriptors the compiler emitted, which is why {@code @JvmOverloads} is + * on those constructors and factories: without it, adding a parameter would delete the descriptor + * an already-compiled consumer is linked against, and the failure would be a + * {@code NoSuchMethodError} at runtime rather than a compile error here. + */ +class MetamodelJavaCompatTest { + + /** DynamicType has no Java-friendly overloads upstream, so every argument is spelled out. */ + private static DomainType type(String name) { + return new DynamicType(name, "", List.of(), List.of(), true); + } + + private static DataDictionary goldenSchema() { + return DataDictionary.fromDomainTypes("golden-schema", List.of(type("Person"), type("Company"))); + } + + @Test + @DisplayName("the four-argument property signature constructor still exists") + void theFourArgumentPropertySignatureConstructorStillExists() { + PropertySignature signature = new PropertySignature( + "age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE); + + assertEquals("age", signature.getName()); + assertEquals(Set.of(), signature.getAliases()); + } + + @Test + @DisplayName("the property signature constructor also takes aliases") + void thePropertySignatureConstructorAlsoTakesAliases() { + PropertySignature signature = new PropertySignature( + "emailAddress", PropertySignature.Kind.VALUE, "string", Cardinality.ONE, Set.of("email")); + + assertEquals(Set.of("email"), signature.getAliases()); + } + + @Test + @DisplayName("the five-argument metamodel version constructor still exists") + void theFiveArgumentMetamodelVersionConstructorStillExists() { + MetamodelVersion version = new MetamodelVersion( + "test", + List.of("Person"), + Map.of("Person", Set.of("Person")), + Map.of("Person", Set.of()), + List.of()); + + assertEquals(List.of("Person"), version.getEntityTypeNames()); + assertEquals(Map.of(), version.getEntityTypeAliases()); + } + + @Test + @DisplayName("the metamodel version constructor also takes aliases") + void theMetamodelVersionConstructorAlsoTakesAliases() { + MetamodelVersion version = new MetamodelVersion( + "test", + List.of("Person"), + Map.of("Person", Set.of("Person")), + Map.of("Person", Set.of()), + List.of(), + Map.of("Person", Set.of("Human"))); + + assertEquals(Set.of("Human"), version.getEntityTypeAliases().get("Person")); + } + + @Test + @DisplayName("the one and two-argument stamping factories still exist") + void theOneAndTwoArgumentStampingFactoriesStillExist() { + MetamodelVersion whole = MetamodelVersion.from(goldenSchema()); + MetamodelVersion governed = MetamodelVersion.from(goldenSchema(), GovernedTypeSelector.ALL); + + assertEquals(whole.getContentHash(), governed.getContentHash()); + assertEquals(List.of("Company", "Person"), whole.getEntityTypeNames()); + } + + @Test + @DisplayName("the stamping factory also takes aliases") + void theStampingFactoryAlsoTakesAliases() { + MetamodelVersion version = MetamodelVersion.from( + goldenSchema(), + GovernedTypeSelector.ALL, + new SchemaAliases(Map.of("Person", Set.of("Human")), Map.of())); + + assertEquals(Set.of("Human"), version.getEntityTypeAliases().get("Person")); + } + + @Test + @DisplayName("the one and two-argument declaration factories still exist") + void theOneAndTwoArgumentDeclarationFactoriesStillExist() { + DeclaredSchema whole = DeclaredSchema.from(goldenSchema()); + DeclaredSchema governed = DeclaredSchema.from(goldenSchema(), GovernedTypeSelector.ALL); + + assertEquals(whole, governed); + assertEquals(Set.of(), whole.getRelationshipTypeNames()); + } + + @Test + @DisplayName("the declaration factory also takes aliases") + void theDeclarationFactoryAlsoTakesAliases() { + DeclaredSchema declared = DeclaredSchema.from( + goldenSchema(), + GovernedTypeSelector.ALL, + new SchemaAliases(Map.of("Person", Set.of("Human")), Map.of())); + + assertEquals(Set.of("Human"), declared.getVersion().getEntityTypeAliases().get("Person")); + } + + @Test + @DisplayName("the no-argument alias constructor exists") + void theNoArgumentAliasConstructorExists() { + assertEquals(Map.of(), new SchemaAliases().getTypeAliases()); + assertEquals(Map.of(), SchemaAliases.NONE.getPropertyAliases()); + } + + @Test + @DisplayName("the collections a stamp hands back refuse mutation from Java") + void theCollectionsAStampHandsBackRefuseMutationFromJava() { + MetamodelVersion version = MetamodelVersion.from( + goldenSchema(), + GovernedTypeSelector.ALL, + new SchemaAliases(Map.of("Person", Set.of("Human")), Map.of())); + + assertTrue(throwsOnMutation(() -> version.getEntityTypeAliases().remove("Person"))); + assertTrue(throwsOnMutation(() -> version.getEntityTypeAliases().get("Person").add("Sneaky"))); + } + + @Test + @DisplayName("the shipped Kotlin default synthetic keeps its descriptor") + void theShippedKotlinDefaultSyntheticKeepsItsDescriptor() throws Exception { + // A Kotlin caller that omits a defaulted argument links against the $default synthetic + // rather than the function itself. DeclaredSchema.from shipped with one defaulted + // parameter, so that synthetic is part of the module's binary surface. Adding a third + // defaulted parameter would have rewritten its descriptor and turned every already + // compiled `DeclaredSchema.from(dictionary)` into a NoSuchMethodError, which is why + // SchemaAliases arrives on a separate overload that requires it. + Class companion = Class.forName("com.embabel.dice.metamodel.DeclaredSchema$Companion"); + + assertNotNull(companion.getDeclaredMethod( + "from$default", + companion, + DataDictionary.class, + GovernedTypeSelector.class, + int.class, + Object.class)); + } + + @Test + @DisplayName("the stamping factories take no defaulted parameters") + void theStampingFactoriesTakeNoDefaultedParameters() throws Exception { + // MetamodelVersion.from shipped as two overloads with no defaults, so it has no $default + // synthetic to preserve. Keeping it that way means the alias overload can never widen one. + Class companion = Class.forName("com.embabel.dice.metamodel.MetamodelVersion$Companion"); + + long defaultSynthetics = Arrays.stream(companion.getDeclaredMethods()) + .filter(method -> method.getName().equals("from$default")) + .count(); + + assertEquals(0, defaultSynthetics); + } + + private static boolean throwsOnMutation(Runnable mutation) { + try { + mutation.run(); + return false; + } catch (UnsupportedOperationException expected) { + return true; + } + } +} diff --git a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/DeclaredSchemaTest.kt b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/DeclaredSchemaTest.kt new file mode 100644 index 00000000..0adf3ecc --- /dev/null +++ b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/DeclaredSchemaTest.kt @@ -0,0 +1,127 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +import com.embabel.agent.core.DataDictionary +import com.embabel.agent.core.DomainTypePropertyDefinition +import com.embabel.agent.core.DynamicType +import com.embabel.agent.core.ValuePropertyDefinition +import org.junit.jupiter.api.Assertions.* +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.assertThrows + +class DeclaredSchemaTest { + + private val company = DynamicType("Company") + private val person = DynamicType( + name = "Person", + ownProperties = listOf(DomainTypePropertyDefinition("worksAt", company)), + ) + private val sighting = DynamicType( + name = "Sighting", + ownProperties = listOf(DomainTypePropertyDefinition("about", person)), + ) + + private fun dictionary() = DataDictionary.fromDomainTypes("app", listOf(person, company, sighting)) + + @Test + fun `declaring everything carries every relationship name`() { + val declared = DeclaredSchema.from(dictionary()) + + assertEquals(setOf("worksAt", "about"), declared.relationshipTypeNames) + assertEquals(MetamodelVersion.from(dictionary()), declared.version) + } + + @Test + fun `bare relationship names follow governance, so they can't disagree with the stamp`() { + // `about` is declared by the ungoverned Sighting, so it is out of both halves. Taking the + // stamp from a governed subset while taking the names from the whole dictionary would + // declare a relationship the stamp never covered. + val declared = DeclaredSchema.from(dictionary(), GovernedTypeSelector { it.name in setOf("Person", "Company") }) + + assertEquals(setOf("worksAt"), declared.relationshipTypeNames) + assertEquals(listOf("Person-[worksAt]->Company"), declared.version.relationshipNames) + assertEquals(listOf("Company", "Person"), declared.version.entityTypeNames) + } + + @Test + fun `a source is just a supplier of the declaration`() { + val source = DeclaredSchemaSource { DeclaredSchema.from(dictionary()) } + + assertEquals(DeclaredSchema.from(dictionary()), source.declare()) + } + + @Test + fun `declaring no aliases matches declaring none explicitly`() { + assertEquals( + DeclaredSchema.from(dictionary()).version.contentHash, + DeclaredSchema.from(dictionary(), GovernedTypeSelector.ALL, SchemaAliases.NONE).version.contentHash, + ) + } + + @Test + fun `declared aliases reach the stamp`() { + val declared = DeclaredSchema.from( + dictionary(), + GovernedTypeSelector.ALL, + SchemaAliases( + typeAliases = mapOf("Person" to setOf("Human")), + propertyAliases = mapOf("Person" to mapOf("worksAt" to setOf("employer"))), + ), + ) + + assertEquals(setOf("Human"), declared.version.entityTypeAliases["Person"]) + assertEquals( + setOf("employer"), + declared.version.entityTypeProperties["Person"]!!.single { it.name == "worksAt" }.aliases, + ) + assertNotEquals(DeclaredSchema.from(dictionary()).version.contentHash, declared.version.contentHash) + } + + @Test + fun `a type alias naming another declared type is refused`() { + val thrown = assertThrows { + DeclaredSchema.from( + dictionary(), + GovernedTypeSelector.ALL, + SchemaAliases(typeAliases = mapOf("Person" to setOf("Company"))), + ) + } + assertTrue(thrown.message!!.contains("Company"), thrown.message) + } + + @Test + fun `aliases on a property name the merge holds two signatures for are refused`() { + // Two same-named Person declarations each carry their own `age`, so the union holds two + // signatures and an old name can't say which of them it meant. + val duplicated = DataDictionary.fromDomainTypes( + "app", + listOf( + DynamicType(name = "Person", ownProperties = listOf(ValuePropertyDefinition("age", type = "string"))), + DynamicType(name = "Person", ownProperties = listOf(ValuePropertyDefinition("age", type = "integer"))), + ), + ) + + val thrown = assertThrows { + DeclaredSchema.from( + duplicated, + GovernedTypeSelector.ALL, + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("age" to setOf("years")))), + ) + } + assertTrue(thrown.message!!.contains("years"), thrown.message) + } +} diff --git a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionStoreTest.kt b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionStoreTest.kt new file mode 100644 index 00000000..e83a0f88 --- /dev/null +++ b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionStoreTest.kt @@ -0,0 +1,105 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +import com.embabel.agent.core.DataDictionary +import com.embabel.agent.core.DynamicType +import org.junit.jupiter.api.Assertions.* +import org.junit.jupiter.api.Test + +/** + * Covers the one piece of behaviour the contract itself ships: the default [findVersion], which a + * backend is free to override with a keyed lookup. A store implementation gets its own tests + * wherever it lives. + */ +class MetamodelVersionStoreTest { + + /** Minimal store honouring the contract: upsert on (schemaName, contentHash), newest first. */ + private class InMemoryVersionStore : MetamodelVersionStore { + + private val saved = mutableListOf() + + override fun saveVersion(version: MetamodelVersion) { + // Idempotent: re-saving an existing version keeps its original position in write order. + if (saved.none { it.schemaName == version.schemaName && it.contentHash == version.contentHash }) { + saved.add(version) + } + } + + override fun latestVersion(schemaName: String): MetamodelVersion? = + versionHistory(schemaName).firstOrNull() + + override fun versionHistory(schemaName: String): List = + saved.filter { it.schemaName == schemaName }.reversed() + } + + private fun version(schemaName: String, vararg typeNames: String): MetamodelVersion = + MetamodelVersion.from( + DataDictionary.fromDomainTypes(schemaName, typeNames.map { DynamicType(name = it) }), + ) + + @Test + fun `findVersion returns the stamp with that hash`() { + val store = InMemoryVersionStore() + val first = version("app", "Person") + val second = version("app", "Person", "Company") + store.saveVersion(first) + store.saveVersion(second) + + assertEquals(first, store.findVersion("app", first.contentHash)) + assertEquals(second, store.findVersion("app", second.contentHash)) + } + + @Test + fun `findVersion is scoped to the schema name`() { + // Two schemas can hold structurally identical versions, because the hash excludes the + // name, so the lookup has to match on both halves of the key. + val store = InMemoryVersionStore() + val mine = version("mine", "Person") + store.saveVersion(mine) + + assertEquals(mine, store.findVersion("mine", mine.contentHash)) + assertNull(store.findVersion("yours", mine.contentHash)) + } + + @Test + fun `findVersion returns null for an unknown hash`() { + val store = InMemoryVersionStore() + store.saveVersion(version("app", "Person")) + + assertNull(store.findVersion("app", "not-a-hash")) + } + + @Test + fun `re-saving a version leaves one record, not two`() { + val store = InMemoryVersionStore() + val v = version("app", "Person") + store.saveVersion(v) + store.saveVersion(v) + + assertEquals(listOf(v), store.versionHistory("app")) + assertEquals(v, store.latestVersion("app")) + } + + @Test + fun `an empty store has no latest version and an empty history`() { + val store = InMemoryVersionStore() + + assertNull(store.latestVersion("app")) + assertEquals(emptyList(), store.versionHistory("app")) + assertNull(store.findVersion("app", "anything")) + } +} diff --git a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt new file mode 100644 index 00000000..329115f1 --- /dev/null +++ b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/MetamodelVersionTest.kt @@ -0,0 +1,1422 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +import com.embabel.agent.core.Cardinality +import com.embabel.agent.core.DataDictionary +import com.embabel.agent.core.DomainType +import com.embabel.agent.core.DomainTypePropertyDefinition +import com.embabel.agent.core.DynamicType +import com.embabel.agent.core.PropertyDefinition +import com.embabel.agent.core.ValuePropertyDefinition +import org.junit.jupiter.api.Assertions.* +import org.junit.jupiter.api.Nested +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.assertThrows + +class MetamodelVersionTest { + + private fun schemaWith(vararg typeNames: String): DataDictionary = + DataDictionary.fromDomainTypes( + "test", + typeNames.map { DynamicType(name = it) }, + ) + + @Nested + inner class ContentHash { + + @Test + fun `identical schemas produce the same hash`() { + val a = MetamodelVersion.from(schemaWith("Person", "Company")) + val b = MetamodelVersion.from(schemaWith("Person", "Company")) + assertEquals(a.contentHash, b.contentHash) + assertTrue(a.hasSameContentAs(b)) + } + + @Test + fun `different type sets produce different hashes`() { + val a = MetamodelVersion.from(schemaWith("Person", "Company")) + val b = MetamodelVersion.from(schemaWith("Person", "Technology")) + assertNotEquals(a.contentHash, b.contentHash) + assertFalse(a.hasSameContentAs(b)) + } + + @Test + fun `type order does not affect hash`() { + val a = MetamodelVersion.from(schemaWith("Company", "Person")) + val b = MetamodelVersion.from(schemaWith("Person", "Company")) + assertEquals(a.contentHash, b.contentHash) + } + + @Test + fun `adding a type changes the hash`() { + val base = MetamodelVersion.from(schemaWith("Person")) + val extended = MetamodelVersion.from(schemaWith("Person", "Company")) + assertNotEquals(base.contentHash, extended.contentHash) + } + + @Test + fun `empty schema has a stable hash`() { + val a = MetamodelVersion.from(schemaWith()) + val b = MetamodelVersion.from(schemaWith()) + assertEquals(a.contentHash, b.contentHash) + } + + @Test + fun `a caller cannot supply the hash — it is always derived from the content`() { + val one = MetamodelVersion( + schemaName = "s", + entityTypeNames = listOf("A"), + entityTypeLabels = mapOf("A" to setOf("A")), + entityTypeProperties = mapOf("A" to emptySet()), + relationshipNames = emptyList(), + ) + val two = MetamodelVersion( + schemaName = "s", + entityTypeNames = listOf("A", "B"), + entityTypeLabels = mapOf("A" to setOf("A"), "B" to setOf("B")), + entityTypeProperties = mapOf("A" to emptySet(), "B" to emptySet()), + relationshipNames = emptyList(), + ) + assertNotEquals(one.contentHash, two.contentHash) + assertFalse(one.hasSameContentAs(two)) + } + + @Test + fun `same types under different schema names produce the same hash`() { + // The schema name is excluded from contentHash, so dev and prod variants of one schema + // compare as equal. + val a = MetamodelVersion.from( + DataDictionary.fromDomainTypes("schema-dev", listOf(DynamicType("Person"))) + ) + val b = MetamodelVersion.from( + DataDictionary.fromDomainTypes("schema-prod", listOf(DynamicType("Person"))) + ) + assertEquals(a.contentHash, b.contentHash) + assertTrue(a.hasSameContentAs(b)) + } + } + + @Nested + inner class GoldenHash { + + /** + * A fixed schema whose every hashed ingredient is spelled out: two type names, one label + * apiece, three properties on one of them, and one relationship. + */ + private fun goldenSchema(): DataDictionary { + val company = DynamicType(name = "Company") + val person = DynamicType( + name = "Person", + ownProperties = listOf( + ValuePropertyDefinition("age"), + ValuePropertyDefinition("email"), + DomainTypePropertyDefinition("worksAt", company), + ), + ) + return DataDictionary.fromDomainTypes("golden-schema", listOf(person, company)) + } + + @Test + fun `the fixture hashes exactly these ingredients`() { + // Guards the golden vector below: if this fails, the fixture changed, not the format. + val version = MetamodelVersion.from(goldenSchema()) + assertEquals(listOf("Company", "Person"), version.entityTypeNames) + assertEquals(setOf("Company"), version.entityTypeLabels["Company"]) + assertEquals(setOf("Person"), version.entityTypeLabels["Person"]) + assertEquals(emptySet(), version.entityTypeProperties["Company"]) + assertEquals( + setOf( + PropertySignature("age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE), + PropertySignature("email", PropertySignature.Kind.VALUE, "string", Cardinality.ONE), + PropertySignature("worksAt", PropertySignature.Kind.REFERENCE, "Company", Cardinality.ONE), + ), + version.entityTypeProperties["Person"], + ) + assertEquals(listOf("Person-[worksAt]->Company"), version.relationshipNames) + } + + @Test + fun `golden vector — the digest of a fixed schema is pinned to a literal`() { + // This literal pins the persisted hash format. contentHash is the store's natural key + // and what extracted data records as the version it was created under, so changing how + // the fingerprint is encoded orphans everything already saved against it. Changing the + // format means changing this literal in the same commit and planning the migration; + // don't "fix" the test by pasting in whatever the new code produces. This vector was + // last regenerated when property signatures (type and cardinality, not just the name) + // went into the encoding, before anything had been persisted against the old form. + assertEquals( + "0a5b5b62c125d8ade5bcd2af5b03e0ec5bcaaf5b0799b7cfe8c16be6e723de00", + MetamodelVersion.from(goldenSchema()).contentHash, + ) + } + + @Test + fun `the golden digest does not depend on the schema name`() { + val renamed = DataDictionary.fromDomainTypes("some-other-name", goldenSchema().domainTypes) + assertEquals( + "0a5b5b62c125d8ade5bcd2af5b03e0ec5bcaaf5b0799b7cfe8c16be6e723de00", + MetamodelVersion.from(renamed).contentHash, + ) + } + + @Test + fun `an alias-free declaration still hashes to the digest pinned before aliases existed`() { + // The literal below was produced by the encoding as it stood before PropertySignature + // carried aliases and MetamodelVersion carried entityTypeAliases. Alias blocks are + // written only when they hold something, so a schema declaring no former names has to + // render the same bytes and keep every hash already recorded against it. All four ways + // of saying "no aliases" have to land on it. + val pinned = "0a5b5b62c125d8ade5bcd2af5b03e0ec5bcaaf5b0799b7cfe8c16be6e723de00" + + assertEquals(pinned, MetamodelVersion.from(goldenSchema()).contentHash) + assertEquals(pinned, MetamodelVersion.from(goldenSchema(), GovernedTypeSelector.ALL).contentHash) + assertEquals( + pinned, + MetamodelVersion.from(goldenSchema(), GovernedTypeSelector.ALL, SchemaAliases.NONE).contentHash, + ) + assertEquals( + pinned, + MetamodelVersion.from( + goldenSchema(), + GovernedTypeSelector.ALL, + SchemaAliases(typeAliases = emptyMap(), propertyAliases = emptyMap()), + ).contentHash, + ) + } + + @Test + fun `rebuilding the golden stamp through the public constructor hashes to the same literal`() { + // The storage mapper reconstructs a stamp field by field rather than from a dictionary. + // Passing an explicitly empty alias map has to reproduce the pinned digest, or a row + // written before aliases existed could never be read back. + val fromDictionary = MetamodelVersion.from(goldenSchema()) + val rebuilt = MetamodelVersion( + schemaName = fromDictionary.schemaName, + entityTypeNames = fromDictionary.entityTypeNames, + entityTypeLabels = fromDictionary.entityTypeLabels, + entityTypeProperties = fromDictionary.entityTypeProperties, + relationshipNames = fromDictionary.relationshipNames, + entityTypeAliases = emptyMap(), + ) + assertEquals( + "0a5b5b62c125d8ade5bcd2af5b03e0ec5bcaaf5b0799b7cfe8c16be6e723de00", + rebuilt.contentHash, + ) + } + } + + @Nested + inner class Relationships { + + /** Same types and properties throughout; only the relationship list varies. */ + private fun versionWithRelationships(vararg relationshipNames: String): MetamodelVersion = + MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Company", "Person"), + entityTypeLabels = mapOf("Company" to setOf("Company"), "Person" to setOf("Person")), + entityTypeProperties = mapOf("Company" to emptySet(), "Person" to emptySet()), + relationshipNames = relationshipNames.toList(), + ) + + @Test + fun `adding a relationship changes the hash`() { + val before = versionWithRelationships() + val after = versionWithRelationships("Person-[WORKS_AT]->Company") + assertNotEquals(before.contentHash, after.contentHash) + assertFalse(before.hasSameContentAs(after)) + } + + @Test + fun `removing one relationship of two changes the hash`() { + val both = versionWithRelationships("Person-[WORKS_AT]->Company", "Person-[FOUNDED]->Company") + val one = versionWithRelationships("Person-[WORKS_AT]->Company") + assertNotEquals(both.contentHash, one.contentHash) + } + + @Test + fun `relationship order does not affect the hash`() { + val a = versionWithRelationships("Person-[WORKS_AT]->Company", "Person-[FOUNDED]->Company") + val b = versionWithRelationships("Person-[FOUNDED]->Company", "Person-[WORKS_AT]->Company") + assertEquals(a.contentHash, b.contentHash) + assertTrue(a.hasSameContentAs(b)) + } + + @Test + fun `dropping a relationship property from a dictionary changes the hash end to end`() { + val company = DynamicType(name = "Company") + val withRelationship = DataDictionary.fromDomainTypes( + "test", + listOf( + DynamicType( + name = "Person", + ownProperties = listOf(DomainTypePropertyDefinition("worksAt", company)), + ), + company, + ), + ) + val withoutRelationship = DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType(name = "Person"), company), + ) + + val before = MetamodelVersion.from(withRelationship) + val after = MetamodelVersion.from(withoutRelationship) + + assertEquals(listOf("Person-[worksAt]->Company"), before.relationshipNames) + assertEquals(emptyList(), after.relationshipNames) + assertNotEquals(before.contentHash, after.contentHash) + } + + @Test + fun `a repeated descriptor is deduplicated, not counted twice`() { + val once = versionWithRelationships("Person-[WORKS_AT]->Company") + val twice = versionWithRelationships("Person-[WORKS_AT]->Company", "Person-[WORKS_AT]->Company") + assertEquals(listOf("Person-[WORKS_AT]->Company"), twice.relationshipNames) + assertEquals(once.contentHash, twice.contentHash) + } + + @Test + fun `splitting a type into two same-named declarations hashes like the merged one`() { + // A DataDictionary can hold two "Person" types that both declare worksAt, which renders + // the same descriptor twice. It is the same schema as one Person declaring it once, so + // it has to be the same hash. + val company = DynamicType(name = "Company") + val worksAt = DomainTypePropertyDefinition("worksAt", company) + val merged = DataDictionary.fromDomainTypes( + "test", + listOf( + DynamicType( + name = "Person", + ownProperties = listOf(ValuePropertyDefinition("age"), worksAt), + ), + company, + ), + ) + val split = DataDictionary.fromDomainTypes( + "test", + listOf( + DynamicType(name = "Person", ownProperties = listOf(ValuePropertyDefinition("age"), worksAt)), + DynamicType(name = "Person", ownProperties = listOf(worksAt)), + company, + ), + ) + + val mergedVersion = MetamodelVersion.from(merged) + val splitVersion = MetamodelVersion.from(split) + + assertEquals(listOf("Person-[worksAt]->Company"), splitVersion.relationshipNames) + assertEquals(mergedVersion.contentHash, splitVersion.contentHash) + assertTrue(mergedVersion.hasSameContentAs(splitVersion)) + } + } + + @Nested + inner class PropertySignatures { + + /** One Person with a single property, described however the test needs it. */ + private fun personWith(property: PropertyDefinition): DataDictionary = + DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType(name = "Person", ownProperties = listOf(property))), + ) + + @Test + fun `changing a property's type produces a different hash`() { + val asString = MetamodelVersion.from(personWith(ValuePropertyDefinition("age", type = "string"))) + val asInteger = MetamodelVersion.from(personWith(ValuePropertyDefinition("age", type = "integer"))) + + assertEquals(asString.entityTypeNames, asInteger.entityTypeNames) + assertEquals(asString.entityTypeLabels, asInteger.entityTypeLabels) + assertNotEquals(asString.contentHash, asInteger.contentHash) + assertFalse(asString.hasSameContentAs(asInteger)) + } + + @Test + fun `changing a property's cardinality produces a different hash`() { + val one = MetamodelVersion.from( + personWith(ValuePropertyDefinition("nickname", cardinality = Cardinality.ONE)) + ) + val list = MetamodelVersion.from( + personWith(ValuePropertyDefinition("nickname", cardinality = Cardinality.LIST)) + ) + + assertNotEquals(one.contentHash, list.contentHash) + } + + @Test + fun `changing a relationship's cardinality produces a different hash`() { + val company = DynamicType(name = "Company") + val one = MetamodelVersion.from( + personWith(DomainTypePropertyDefinition("worksAt", company, Cardinality.ONE)) + ) + val many = MetamodelVersion.from( + personWith(DomainTypePropertyDefinition("worksAt", company, Cardinality.LIST)) + ) + + // The rendered descriptor is identical; only the cardinality behind it moved. + assertEquals(one.relationshipNames, many.relationshipNames) + assertNotEquals(one.contentHash, many.contentHash) + } + + @Test + fun `retargeting a relationship produces a different hash`() { + val toCompany = MetamodelVersion.from( + personWith(DomainTypePropertyDefinition("worksAt", DynamicType("Company"))) + ) + val toCharity = MetamodelVersion.from( + personWith(DomainTypePropertyDefinition("worksAt", DynamicType("Charity"))) + ) + + assertNotEquals(toCompany.contentHash, toCharity.contentHash) + } + + @Test + fun `a value property and a reference of the same name hash differently`() { + // "worksAt: string" and "worksAt -> Company" are different schemas even though the + // property name, the declared type name, and the cardinality all read the same. + val asValue = MetamodelVersion.from(personWith(ValuePropertyDefinition("worksAt", type = "Company"))) + val asReference = MetamodelVersion.from( + personWith(DomainTypePropertyDefinition("worksAt", DynamicType("Company"))) + ) + + assertNotEquals(asValue.contentHash, asReference.contentHash) + } + + @Test + fun `descriptions and metadata are not part of the signature`() { + // They steer extraction, but they don't change what the graph can hold. + val plain = MetamodelVersion.from(personWith(ValuePropertyDefinition("age"))) + val documented = MetamodelVersion.from( + personWith( + ValuePropertyDefinition( + name = "age", + description = "how many years the person has been alive", + metadata = mapOf("predicate" to "is aged"), + ) + ) + ) + + assertEquals(plain.contentHash, documented.contentHash) + } + + @Test + fun `the signature records name, kind, type and cardinality`() { + val version = MetamodelVersion.from( + personWith(ValuePropertyDefinition("nicknames", type = "string", cardinality = Cardinality.SET)) + ) + + assertEquals( + setOf(PropertySignature("nicknames", PropertySignature.Kind.VALUE, "string", Cardinality.SET)), + version.entityTypeProperties["Person"], + ) + } + } + + @Nested + inner class Immutability { + + private fun version(): MetamodelVersion = MetamodelVersion.from( + DataDictionary.fromDomainTypes( + "test", + listOf( + DynamicType(name = "Person", ownProperties = listOf(ValuePropertyDefinition("age"))), + DynamicType("Company"), + ), + ), + ) + + @Test + fun `the collections a stamp hands back cannot be mutated`() { + // Kotlin's read-only types are a compile-time promise; a Java caller sees plain + // java.util collections through the getters. These have to refuse at runtime, or a + // caller could reshape a stamp out from under its own precomputed hash. + val version = version() + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeNames as MutableList).add("Sneaky") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.relationshipNames as MutableList).add("Sneaky") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeLabels as MutableMap>).remove("Person") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeLabels["Person"] as MutableSet).add("Sneaky") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeProperties as MutableMap>).remove("Person") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeProperties["Person"] as MutableSet).clear() + } + } + + @Test + fun `mutating what the caller passed in does not change the stamp`() { + val names = mutableListOf("Person") + val labels = mutableMapOf("Person" to mutableSetOf("Person")) + val properties = mutableMapOf( + "Person" to mutableSetOf( + PropertySignature("age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE) + ), + ) + val relationships = mutableListOf("Person-[worksAt]->Company") + + val version = MetamodelVersion("test", names, labels, properties, relationships) + val hashAtConstruction = version.contentHash + + names.add("Company") + labels["Person"]!!.add("Agent") + properties["Person"]!!.clear() + relationships.clear() + + assertEquals(listOf("Person"), version.entityTypeNames) + assertEquals(setOf("Person"), version.entityTypeLabels["Person"]) + assertEquals(1, version.entityTypeProperties["Person"]!!.size) + assertEquals(listOf("Person-[worksAt]->Company"), version.relationshipNames) + assertEquals(hashAtConstruction, version.contentHash) + } + } + + @Nested + inner class StructuralConsistency { + + @Test + fun `labels keyed by a type that is not listed are rejected`() { + // Only listed types are walked when hashing, so accepting this would let two stamps + // with different labels share a content hash, and the store's natural key with it. + val thrown = assertThrows { + MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = mapOf("Person" to setOf("Person"), "Ghost" to setOf("Ghost")), + entityTypeProperties = emptyMap(), + relationshipNames = emptyList(), + ) + } + assertTrue(thrown.message!!.contains("entityTypeLabels"), thrown.message) + assertTrue(thrown.message!!.contains("Ghost"), thrown.message) + } + + @Test + fun `properties keyed by a type that is not listed are rejected`() { + val thrown = assertThrows { + MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = mapOf( + "Ghost" to setOf( + PropertySignature("age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE) + ), + ), + relationshipNames = emptyList(), + ) + } + assertTrue(thrown.message!!.contains("entityTypeProperties"), thrown.message) + assertTrue(thrown.message!!.contains("Ghost"), thrown.message) + } + + @Test + fun `entity type names are sorted and deduplicated on the way in`() { + val version = MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person", "Company", "Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = emptyMap(), + relationshipNames = emptyList(), + ) + assertEquals(listOf("Company", "Person"), version.entityTypeNames) + } + } + + @Nested + inner class VersionMetadata { + + @Test + fun `entity type names are sorted`() { + val version = MetamodelVersion.from(schemaWith("Zebra", "Apple", "Mango")) + assertEquals(listOf("Apple", "Mango", "Zebra"), version.entityTypeNames) + } + + @Test + fun `schema name is captured`() { + val dict = DataDictionary.fromDomainTypes("my-schema", listOf(DynamicType("Person"))) + val version = MetamodelVersion.from(dict) + assertEquals("my-schema", version.schemaName) + } + + @Test + fun `per-type label sets are captured including inherited labels`() { + val dict = DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType(name = "Person", parents = listOf(DynamicType(name = "Agent")))), + ) + val version = MetamodelVersion.from(dict) + assertEquals(setOf("Person", "Agent"), version.entityTypeLabels["Person"]) + } + + @Test + fun `per-type property sets are captured`() { + val dict = DataDictionary.fromDomainTypes( + "test", + listOf( + DynamicType( + name = "Person", + ownProperties = listOf(ValuePropertyDefinition("age"), ValuePropertyDefinition("email")), + ), + ), + ) + val version = MetamodelVersion.from(dict) + assertEquals( + setOf("age", "email"), + version.entityTypeProperties["Person"]!!.map { it.name }.toSet(), + ) + } + + @Test + fun `same-named types with different shapes are merged, not dropped`() { + // A DataDictionary can hold two "Person" types with different shapes. The fingerprint + // unions both. Keeping only the last would drop a label or property from the hash, and + // its later removal would go undetected. + val dict = DataDictionary.fromDomainTypes( + "test", + listOf( + DynamicType( + name = "Person", + parents = listOf(DynamicType(name = "Agent")), + ownProperties = listOf(ValuePropertyDefinition("age")), + ), + DynamicType( + name = "Person", + parents = listOf(DynamicType(name = "Robot")), + ownProperties = listOf(ValuePropertyDefinition("email")), + ), + ), + ) + val version = MetamodelVersion.from(dict) + assertEquals(setOf("Person", "Agent", "Robot"), version.entityTypeLabels["Person"]) + assertEquals( + setOf("age", "email"), + version.entityTypeProperties["Person"]!!.map { it.name }.toSet(), + ) + // The name is deduped in the sorted name list. + assertEquals(listOf("Person"), version.entityTypeNames) + } + } + + @Nested + inner class Labels { + + /** Same type name with a different parent, so the label set differs while the name doesn't. */ + private fun personWithParent(parent: String): DataDictionary = + DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType(name = "Person", parents = listOf(DynamicType(name = parent)))), + ) + + @Test + fun `a label-only change produces a different hash`() { + val a = MetamodelVersion.from(personWithParent("Agent")) + val b = MetamodelVersion.from(personWithParent("Actor")) + // The type name set is identical; only the label sets differ. + assertEquals(a.entityTypeNames, b.entityTypeNames) + assertNotEquals(a.contentHash, b.contentHash) + assertFalse(a.hasSameContentAs(b)) + } + + @Test + fun `identical label sets produce the same hash`() { + val a = MetamodelVersion.from(personWithParent("Agent")) + val b = MetamodelVersion.from(personWithParent("Agent")) + assertEquals(a.contentHash, b.contentHash) + } + } + + @Nested + inner class Properties { + + /** Same type name, but a different property set. */ + private fun personWithProperties(vararg props: String): DataDictionary = + DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType(name = "Person", ownProperties = props.map { ValuePropertyDefinition(it) })), + ) + + @Test + fun `a property-only change produces a different hash`() { + val a = MetamodelVersion.from(personWithProperties("age")) + val b = MetamodelVersion.from(personWithProperties("age", "email")) + assertEquals(a.entityTypeNames, b.entityTypeNames) + assertEquals(a.entityTypeLabels, b.entityTypeLabels) + assertNotEquals(a.contentHash, b.contentHash) + assertFalse(a.hasSameContentAs(b)) + } + + @Test + fun `identical property sets produce the same hash`() { + val a = MetamodelVersion.from(personWithProperties("age", "email")) + val b = MetamodelVersion.from(personWithProperties("email", "age")) + assertEquals(a.contentHash, b.contentHash) + } + + @Test + fun `a property name containing the set delimiter does not collide with a split set`() { + // ["a;b"] and ["a", "b"] are different property sets. A delimiter-joined encoding would + // serialise both as "a;b;" and hash them identically, hiding a lossy schema change. + // Length-prefixed encoding keeps them distinct. + val joined = MetamodelVersion.from(personWithProperties("a;b")) + val split = MetamodelVersion.from(personWithProperties("a", "b")) + assertNotEquals(joined.contentHash, split.contentHash) + assertFalse(joined.hasSameContentAs(split)) + } + } + + @Nested + inner class GovernedSubset { + + private val governed = GovernedTypeSelector { it.name in setOf("Person", "Company") } + + private fun dictionaryOf(vararg types: DomainType): DataDictionary = + DataDictionary.fromDomainTypes("test", types.toList()) + + @Test + fun `the default selector governs everything`() { + val dict = dictionaryOf(DynamicType("Person"), DynamicType("Company")) + assertEquals( + MetamodelVersion.from(dict).contentHash, + MetamodelVersion.from(dict, GovernedTypeSelector.ALL).contentHash, + ) + assertEquals(listOf("Company", "Person"), MetamodelVersion.from(dict, GovernedTypeSelector.ALL).entityTypeNames) + } + + @Test + fun `adding an ungoverned type leaves the hash alone`() { + // Per-type governance exists so that extraction proposing a new exploratory type + // doesn't fill the version history with stamps nobody chose. + val before = dictionaryOf(DynamicType("Person"), DynamicType("Company")) + val after = dictionaryOf(DynamicType("Person"), DynamicType("Company"), DynamicType("Sighting")) + + assertEquals( + MetamodelVersion.from(before, governed).contentHash, + MetamodelVersion.from(after, governed).contentHash, + ) + } + + @Test + fun `adding a governed type changes the hash`() { + val before = dictionaryOf(DynamicType("Person")) + val after = dictionaryOf(DynamicType("Person"), DynamicType("Company")) + + assertNotEquals( + MetamodelVersion.from(before, governed).contentHash, + MetamodelVersion.from(after, governed).contentHash, + ) + } + + @Test + fun `reshaping an ungoverned type leaves the hash alone`() { + // Labels and properties on an ungoverned type are excluded along with its name. + val plain = dictionaryOf(DynamicType("Person"), DynamicType("Sighting")) + val reshaped = dictionaryOf( + DynamicType("Person"), + DynamicType( + name = "Sighting", + parents = listOf(DynamicType("Observation")), + ownProperties = listOf(ValuePropertyDefinition("seenAt")), + ), + ) + + assertEquals( + MetamodelVersion.from(plain, governed).contentHash, + MetamodelVersion.from(reshaped, governed).contentHash, + ) + } + + @Test + fun `reshaping a governed type changes the hash`() { + val plain = dictionaryOf(DynamicType("Person"), DynamicType("Sighting")) + val reshaped = dictionaryOf( + DynamicType(name = "Person", ownProperties = listOf(ValuePropertyDefinition("age"))), + DynamicType("Sighting"), + ) + + assertNotEquals( + MetamodelVersion.from(plain, governed).contentHash, + MetamodelVersion.from(reshaped, governed).contentHash, + ) + } + + @Test + fun `only governed types are stamped`() { + val version = MetamodelVersion.from( + dictionaryOf(DynamicType("Person"), DynamicType("Company"), DynamicType("Sighting")), + governed, + ) + assertEquals(listOf("Company", "Person"), version.entityTypeNames) + assertNull(version.entityTypeLabels["Sighting"]) + assertNull(version.entityTypeProperties["Sighting"]) + } + + @Test + fun `a relationship declared by an ungoverned type is left out`() { + val person = DynamicType("Person") + val withSighting = dictionaryOf( + person, + DynamicType(name = "Sighting", ownProperties = listOf(DomainTypePropertyDefinition("about", person))), + ) + val withoutSighting = dictionaryOf(person) + + val version = MetamodelVersion.from(withSighting, governed) + assertEquals(emptyList(), version.relationshipNames) + assertEquals(MetamodelVersion.from(withoutSighting, governed).contentHash, version.contentHash) + } + + @Test + fun `a governed type's relationship to an ungoverned type stays in the stamp`() { + // The relationship belongs to the type that declares it. Person saying it has a + // `spotted` Sighting is part of Person's declared shape, whether or not Sighting is + // itself governed, and whether or not Sighting is in the dictionary at all. + val sighting = DynamicType("Sighting") + val person = DynamicType( + name = "Person", + ownProperties = listOf(DomainTypePropertyDefinition("spotted", sighting)), + ) + + val version = MetamodelVersion.from(dictionaryOf(person, sighting), governed) + assertEquals(listOf("Person-[spotted]->Sighting"), version.relationshipNames) + // Listing the ungoverned type in the dictionary therefore changes nothing. + assertEquals(MetamodelVersion.from(dictionaryOf(person), governed).contentHash, version.contentHash) + } + + @Test + fun `dropping a governed type's relationship changes the hash`() { + val sighting = DynamicType("Sighting") + val with = dictionaryOf( + DynamicType(name = "Person", ownProperties = listOf(DomainTypePropertyDefinition("spotted", sighting))), + sighting, + ) + val without = dictionaryOf(DynamicType("Person"), sighting) + + assertNotEquals( + MetamodelVersion.from(with, governed).contentHash, + MetamodelVersion.from(without, governed).contentHash, + ) + } + + @Test + fun `governing nothing stamps an empty schema`() { + val version = MetamodelVersion.from( + dictionaryOf(DynamicType("Person"), DynamicType("Company")), + GovernedTypeSelector { false }, + ) + assertEquals(emptyList(), version.entityTypeNames) + assertEquals(emptyList(), version.relationshipNames) + assertEquals( + MetamodelVersion.from(DataDictionary.fromDomainTypes("test", emptyList())).contentHash, + version.contentHash, + ) + } + + @Test + fun `the schema name is still captured when only a subset is governed`() { + val version = MetamodelVersion.from( + DataDictionary.fromDomainTypes("my-schema", listOf(DynamicType("Person"), DynamicType("Sighting"))), + governed, + ) + assertEquals("my-schema", version.schemaName) + } + } + + @Nested + inner class Aliases { + + private fun personWith(vararg properties: PropertyDefinition): DataDictionary = + DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType(name = "Person", ownProperties = properties.toList())), + ) + + private fun stamp(dictionary: DataDictionary, aliases: SchemaAliases): MetamodelVersion = + MetamodelVersion.from(dictionary, GovernedTypeSelector.ALL, aliases) + + @Test + fun `a declared property alias lands on the signature`() { + val version = stamp( + personWith(ValuePropertyDefinition("emailAddress")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email")))), + ) + + assertEquals( + setOf( + PropertySignature( + "emailAddress", + PropertySignature.Kind.VALUE, + "string", + Cardinality.ONE, + setOf("email"), + ), + ), + version.entityTypeProperties["Person"], + ) + } + + @Test + fun `declaring a property alias changes the hash`() { + val plain = stamp(personWith(ValuePropertyDefinition("emailAddress")), SchemaAliases.NONE) + val aliased = stamp( + personWith(ValuePropertyDefinition("emailAddress")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email")))), + ) + + assertNotEquals(plain.contentHash, aliased.contentHash) + assertFalse(plain.hasSameContentAs(aliased)) + } + + @Test + fun `the order aliases are declared in does not affect the hash`() { + val forwards = stamp( + personWith(ValuePropertyDefinition("emailAddress")), + SchemaAliases( + propertyAliases = mapOf("Person" to mapOf("emailAddress" to linkedSetOf("email", "contact"))), + ), + ) + val backwards = stamp( + personWith(ValuePropertyDefinition("emailAddress")), + SchemaAliases( + propertyAliases = mapOf("Person" to mapOf("emailAddress" to linkedSetOf("contact", "email"))), + ), + ) + + assertEquals(forwards.contentHash, backwards.contentHash) + } + + @Test + fun `different alias sets on one property hash differently`() { + val one = stamp( + personWith(ValuePropertyDefinition("emailAddress")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email")))), + ) + val two = stamp( + personWith(ValuePropertyDefinition("emailAddress")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email", "contact")))), + ) + + assertNotEquals(one.contentHash, two.contentHash) + } + + @Test + fun `an alias containing the block delimiter does not collide with a split set`() { + // Same reasoning as the property-name case: alias entries are length-prefixed, so + // ["a;b"] and ["a", "b"] can't serialise to the same bytes. + val joined = stamp( + personWith(ValuePropertyDefinition("emailAddress")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("a;b")))), + ) + val split = stamp( + personWith(ValuePropertyDefinition("emailAddress")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("a", "b")))), + ) + + assertNotEquals(joined.contentHash, split.contentHash) + } + + @Test + fun `an alias for a property the type doesn't have changes nothing`() { + val plain = stamp(personWith(ValuePropertyDefinition("age")), SchemaAliases.NONE) + val stale = stamp( + personWith(ValuePropertyDefinition("age")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("retired" to setOf("gone")))), + ) + + assertEquals(plain.contentHash, stale.contentHash) + } + + @Test + fun `an alias equal to the property's own name is kept and hashed`() { + // It matches nothing at diff time — nothing looks data up by property name — so it is + // inert there. It is still part of the signature, so it moves the hash. + val plain = stamp(personWith(ValuePropertyDefinition("age")), SchemaAliases.NONE) + val selfAliased = stamp( + personWith(ValuePropertyDefinition("age")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("age" to setOf("age")))), + ) + + assertEquals(setOf("age"), selfAliased.entityTypeProperties["Person"]!!.single().aliases) + assertNotEquals(plain.contentHash, selfAliased.contentHash) + } + + @Test + fun `a declared type alias is carried and changes the hash`() { + val plain = stamp(personWith(), SchemaAliases.NONE) + val aliased = stamp(personWith(), SchemaAliases(typeAliases = mapOf("Person" to setOf("Human")))) + + assertEquals(mapOf("Person" to setOf("Human")), aliased.entityTypeAliases) + assertEquals(emptyMap>(), plain.entityTypeAliases) + assertNotEquals(plain.contentHash, aliased.contentHash) + } + + @Test + fun `type alias order does not affect the hash`() { + val forwards = stamp(personWith(), SchemaAliases(typeAliases = mapOf("Person" to linkedSetOf("Human", "Actor")))) + val backwards = stamp(personWith(), SchemaAliases(typeAliases = mapOf("Person" to linkedSetOf("Actor", "Human")))) + + assertEquals(forwards.contentHash, backwards.contentHash) + } + + @Test + fun `a type alias and a property alias of the same name hash differently`() { + // The two blocks carry different tags and sit in different places, so declaring "old" + // as a former type name is a different schema from declaring it as a former property + // name. + val asTypeAlias = stamp( + personWith(ValuePropertyDefinition("age")), + SchemaAliases(typeAliases = mapOf("Person" to setOf("old"))), + ) + val asPropertyAlias = stamp( + personWith(ValuePropertyDefinition("age")), + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("age" to setOf("old")))), + ) + + assertNotEquals(asTypeAlias.contentHash, asPropertyAlias.contentHash) + } + + @Test + fun `aliases accumulate across successive renames`() { + val version = stamp( + DataDictionary.fromDomainTypes("test", listOf(DynamicType("C"))), + SchemaAliases(typeAliases = mapOf("C" to setOf("A", "B"))), + ) + + assertEquals(setOf("A", "B"), version.entityTypeAliases["C"]) + } + + @Test + fun `a type may list its own name, which is what a rename and back leaves behind`() { + // A renamed to B and back to A accumulates {A, B}, so the alias set holds the current + // name. The reuse guard is about other types' names. + val version = stamp( + DataDictionary.fromDomainTypes("test", listOf(DynamicType("A"))), + SchemaAliases(typeAliases = mapOf("A" to setOf("A", "B"))), + ) + + assertEquals(setOf("A", "B"), version.entityTypeAliases["A"]) + } + + @Test + fun `aliases for an ungoverned type are dropped`() { + // Everything else about an ungoverned type is invisible to the stamp; aliases follow. + val dictionary = DataDictionary.fromDomainTypes( + "test", + listOf( + DynamicType("Person"), + DynamicType(name = "Sighting", ownProperties = listOf(ValuePropertyDefinition("seenAt"))), + ), + ) + val governed = GovernedTypeSelector { it.name == "Person" } + + val plain = MetamodelVersion.from(dictionary, governed) + val aliased = MetamodelVersion.from( + dictionary, + governed, + SchemaAliases( + typeAliases = mapOf("Sighting" to setOf("Observation")), + propertyAliases = mapOf("Sighting" to mapOf("seenAt" to setOf("spottedAt"))), + ), + ) + + assertEquals(emptyMap>(), aliased.entityTypeAliases) + assertEquals(plain.contentHash, aliased.contentHash) + } + + @Test + fun `an explicitly empty alias set is dropped rather than hashed`() { + val plain = stamp(personWith(ValuePropertyDefinition("age")), SchemaAliases.NONE) + val declaredEmpty = stamp( + personWith(ValuePropertyDefinition("age")), + SchemaAliases( + typeAliases = mapOf("Person" to emptySet()), + propertyAliases = mapOf("Person" to mapOf("age" to emptySet())), + ), + ) + + assertEquals(emptyMap>(), declaredEmpty.entityTypeAliases) + assertEquals(plain.contentHash, declaredEmpty.contentHash) + } + } + + @Nested + inner class SignatureOrdering { + + private fun signature(name: String, aliases: Set = emptySet()): PropertySignature = + PropertySignature(name, PropertySignature.Kind.VALUE, "string", Cardinality.ONE, aliases) + + @Test + fun `aliases break ties only after name, kind, type and cardinality`() { + val aliasedAge = signature("age", setOf("zzz")) + val plainEmail = signature("email") + + // The name still decides, whatever the aliases say. + assertTrue(aliasedAge < plainEmail) + + val aliasedString = PropertySignature( + "age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE, setOf("zzz"), + ) + val plainInteger = PropertySignature( + "age", PropertySignature.Kind.VALUE, "integer", Cardinality.ONE, + ) + assertTrue(plainInteger < aliasedString) + } + + @Test + fun `signatures differing only in aliases sort deterministically`() { + val none = signature("age") + val one = signature("age", setOf("b")) + val two = signature("age", setOf("a1", "b")) + + assertTrue(none < two) + assertTrue(two < one) + assertEquals(listOf(none, two, one), listOf(one, none, two).sorted()) + assertEquals(listOf(none, two, one), listOf(two, one, none).sorted()) + } + + @Test + fun `alias order inside the set does not change the ordering`() { + val forwards = signature("age", linkedSetOf("a", "b")) + val backwards = signature("age", linkedSetOf("b", "a")) + + assertEquals(0, forwards.compareTo(backwards)) + } + } + + @Nested + inner class AliasGuards { + + private fun personTwice(vararg propertyTypes: String): DataDictionary = + DataDictionary.fromDomainTypes( + "test", + propertyTypes.map { propertyType -> + DynamicType( + name = "Person", + ownProperties = listOf(ValuePropertyDefinition("age", type = propertyType)), + ) + }, + ) + + @Test + fun `type aliases keyed by a type that is not listed are rejected`() { + val thrown = assertThrows { + MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = emptyMap(), + relationshipNames = emptyList(), + entityTypeAliases = mapOf("Ghost" to setOf("Spectre")), + ) + } + assertTrue(thrown.message!!.contains("entityTypeAliases"), thrown.message) + assertTrue(thrown.message!!.contains("Ghost"), thrown.message) + } + + @Test + fun `an empty type alias set is rejected`() { + // An entry with no former names in it hashes differently from having no entry at all, + // while meaning the same thing, so two stamps of one schema could land on two keys. + val thrown = assertThrows { + MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = emptyMap(), + relationshipNames = emptyList(), + entityTypeAliases = mapOf("Person" to emptySet()), + ) + } + assertTrue(thrown.message!!.contains("empty alias sets"), thrown.message) + assertTrue(thrown.message!!.contains("Person"), thrown.message) + } + + @Test + fun `a declared type name in another type's alias set is rejected by the constructor`() { + val thrown = assertThrows { + MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Human", "Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = emptyMap(), + relationshipNames = emptyList(), + entityTypeAliases = mapOf("Human" to setOf("Person")), + ) + } + assertTrue(thrown.message!!.contains("Human"), thrown.message) + assertTrue(thrown.message!!.contains("Person"), thrown.message) + assertTrue(thrown.message!!.contains("Retire the alias"), thrown.message) + } + + @Test + fun `a declared type name in another type's alias set is rejected at stamping`() { + val dictionary = DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType("Human"), DynamicType("Person")), + ) + + val thrown = assertThrows { + MetamodelVersion.from( + dictionary, + GovernedTypeSelector.ALL, + SchemaAliases(typeAliases = mapOf("Human" to setOf("Person"))), + ) + } + assertTrue(thrown.message!!.contains("Person"), thrown.message) + } + + @Test + fun `reusing the name of an ungoverned type is allowed, because the stamp never sees it`() { + val dictionary = DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType("Human"), DynamicType("Person")), + ) + + val version = MetamodelVersion.from( + dictionary, + GovernedTypeSelector { it.name == "Human" }, + SchemaAliases(typeAliases = mapOf("Human" to setOf("Person"))), + ) + assertEquals(setOf("Person"), version.entityTypeAliases["Human"]) + } + + @Test + fun `aliases on a property name with more than one signature are rejected by the constructor`() { + val thrown = assertThrows { + MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = mapOf( + "Person" to setOf( + PropertySignature( + "age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE, setOf("years"), + ), + PropertySignature( + "age", PropertySignature.Kind.VALUE, "integer", Cardinality.ONE, setOf("years"), + ), + ), + ), + relationshipNames = emptyList(), + ) + } + assertTrue(thrown.message!!.contains("age"), thrown.message) + assertTrue(thrown.message!!.contains("years"), thrown.message) + assertTrue(thrown.message!!.contains("Retire the alias"), thrown.message) + } + + @Test + fun `aliases on a property name with more than one signature are rejected at stamping`() { + // Two same-named Person declarations each carry their own `age`, so the union holds two + // signatures for one name and an old name can't say which it meant. + val thrown = assertThrows { + MetamodelVersion.from( + personTwice("string", "integer"), + GovernedTypeSelector.ALL, + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("age" to setOf("years")))), + ) + } + assertTrue(thrown.message!!.contains("age"), thrown.message) + assertTrue(thrown.message!!.contains("years"), thrown.message) + } + + @Test + fun `a duplicated property name with no aliases declared is fine`() { + val version = MetamodelVersion.from(personTwice("string", "integer")) + assertEquals(2, version.entityTypeProperties["Person"]!!.size) + } + + @Test + fun `aliases on a single-signature property survive a duplicate elsewhere on the type`() { + val dictionary = DataDictionary.fromDomainTypes( + "test", + listOf( + DynamicType( + name = "Person", + ownProperties = listOf( + ValuePropertyDefinition("age", type = "string"), + ValuePropertyDefinition("emailAddress"), + ), + ), + DynamicType( + name = "Person", + ownProperties = listOf(ValuePropertyDefinition("age", type = "integer")), + ), + ), + ) + + val version = MetamodelVersion.from( + dictionary, + GovernedTypeSelector.ALL, + SchemaAliases(propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email")))), + ) + + assertEquals( + setOf("email"), + version.entityTypeProperties["Person"]!!.single { it.name == "emailAddress" }.aliases, + ) + } + } + + @Nested + inner class AliasImmutability { + + @Test + fun `the alias collections a stamp hands back cannot be mutated`() { + val version = MetamodelVersion.from( + DataDictionary.fromDomainTypes( + "test", + listOf(DynamicType(name = "Person", ownProperties = listOf(ValuePropertyDefinition("age")))), + ), + GovernedTypeSelector.ALL, + SchemaAliases( + typeAliases = mapOf("Person" to setOf("Human")), + propertyAliases = mapOf("Person" to mapOf("age" to setOf("years"))), + ), + ) + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeAliases as MutableMap>).remove("Person") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeAliases["Person"] as MutableSet).add("Sneaky") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeProperties["Person"]!!.single().aliases as MutableSet).add("Sneaky") + } + } + + @Test + fun `mutating the alias sets the caller passed in does not change the stamp`() { + val typeAliases = mutableSetOf("Human") + val signatureAliases = mutableSetOf("years") + val version = MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = mapOf( + "Person" to setOf( + PropertySignature( + "age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE, signatureAliases, + ), + ), + ), + relationshipNames = emptyList(), + entityTypeAliases = mapOf("Person" to typeAliases), + ) + val hashAtConstruction = version.contentHash + + typeAliases.add("Actor") + signatureAliases.add("yearsOld") + + assertEquals(setOf("Human"), version.entityTypeAliases["Person"]) + assertEquals(setOf("years"), version.entityTypeProperties["Person"]!!.single().aliases) + assertEquals(hashAtConstruction, version.contentHash) + } + + @Test + fun `filling in an alias set that was empty at construction does not change the stamp`() { + // The empty case is the dangerous one. A signature built with an empty mutable set that + // the stamp stored by reference would change its own hashCode when the caller added an + // alias, leaving it unfindable in the hash-based set holding it and disagreeing with a + // contentHash computed while it looked alias-free. + val aliases = mutableSetOf() + val signature = PropertySignature( + "age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE, aliases, + ) + val version = MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = mapOf("Person" to setOf(signature)), + relationshipNames = emptyList(), + ) + val hashAtConstruction = version.contentHash + + aliases.add("years") + + val stored = version.entityTypeProperties["Person"]!! + assertEquals(emptySet(), stored.single().aliases) + assertEquals(hashAtConstruction, version.contentHash) + + // The signature is still findable under the identity it was hashed with, so nothing has + // shifted position in the set that holds it. + assertTrue( + stored.contains( + PropertySignature("age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE) + ), + ) + + // And the stamp still hashes as the alias-free schema it was built from. + val neverAliased = MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = mapOf( + "Person" to setOf( + PropertySignature("age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE) + ), + ), + relationshipNames = emptyList(), + ) + assertEquals(neverAliased.contentHash, version.contentHash) + } + + @Test + fun `an initially empty alias set is replaced by an immutable one`() { + val signature = PropertySignature( + "age", PropertySignature.Kind.VALUE, "string", Cardinality.ONE, mutableSetOf(), + ) + val version = MetamodelVersion( + schemaName = "test", + entityTypeNames = listOf("Person"), + entityTypeLabels = emptyMap(), + entityTypeProperties = mapOf("Person" to setOf(signature)), + relationshipNames = emptyList(), + ) + + @Suppress("UNCHECKED_CAST") + assertThrows { + (version.entityTypeProperties["Person"]!!.single().aliases as MutableSet) + .add("Sneaky") + } + } + } +} diff --git a/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/SchemaAliasesTest.kt b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/SchemaAliasesTest.kt new file mode 100644 index 00000000..23450d8d --- /dev/null +++ b/dice-metamodel/src/test/kotlin/com/embabel/dice/metamodel/SchemaAliasesTest.kt @@ -0,0 +1,132 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.metamodel + +import org.junit.jupiter.api.Assertions.* +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.assertThrows + +class SchemaAliasesTest { + + @Test + fun `NONE declares nothing`() { + assertEquals(emptyMap>(), SchemaAliases.NONE.typeAliases) + assertEquals(emptyMap>>(), SchemaAliases.NONE.propertyAliases) + assertEquals(SchemaAliases(), SchemaAliases.NONE) + } + + @Test + fun `an empty type alias set is dropped`() { + // Saying a type has no former names is the same as saying nothing about it, and an empty + // entry would otherwise be refused by the stamp's guard. + val aliases = SchemaAliases(typeAliases = mapOf("Person" to emptySet(), "Company" to setOf("Corp"))) + + assertEquals(mapOf("Company" to setOf("Corp")), aliases.typeAliases) + } + + @Test + fun `an empty property alias set is dropped, and a type left with none goes with it`() { + val aliases = SchemaAliases( + propertyAliases = mapOf( + "Person" to mapOf("age" to emptySet(), "emailAddress" to setOf("email")), + "Company" to mapOf("name" to emptySet()), + ), + ) + + assertEquals(mapOf("Person" to mapOf("emailAddress" to setOf("email"))), aliases.propertyAliases) + } + + @Test + fun `propertyAliasesFor answers empty for anything undeclared`() { + val aliases = SchemaAliases( + propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email"))), + ) + + assertEquals(setOf("email"), aliases.propertyAliasesFor("Person", "emailAddress")) + assertEquals(emptySet(), aliases.propertyAliasesFor("Person", "age")) + assertEquals(emptySet(), aliases.propertyAliasesFor("Company", "emailAddress")) + } + + @Test + fun `alias names are case-sensitive`() { + // LLM extraction drifts on case. Folding it here would pair two names nobody said were the + // same one. + val aliases = SchemaAliases(typeAliases = mapOf("Person" to setOf("Human"))) + + assertNotEquals(aliases, SchemaAliases(typeAliases = mapOf("Person" to setOf("human")))) + assertNotEquals(aliases, SchemaAliases(typeAliases = mapOf("person" to setOf("Human")))) + } + + @Test + fun `mutating what the caller passed in does not change the declaration`() { + val types = mutableMapOf("Person" to mutableSetOf("Human")) + val properties = mutableMapOf("Person" to mutableMapOf("emailAddress" to mutableSetOf("email"))) + + val aliases = SchemaAliases(types, properties) + + types["Ghost"] = mutableSetOf("Spectre") + types["Person"]!!.add("Actor") + properties["Person"]!!["age"] = mutableSetOf("years") + properties["Person"]!!["emailAddress"]!!.add("contact") + + assertEquals(mapOf("Person" to setOf("Human")), aliases.typeAliases) + assertEquals(mapOf("Person" to mapOf("emailAddress" to setOf("email"))), aliases.propertyAliases) + } + + @Test + fun `the collections a declaration hands back cannot be mutated`() { + val aliases = SchemaAliases( + typeAliases = mapOf("Person" to setOf("Human")), + propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email"))), + ) + + @Suppress("UNCHECKED_CAST") + assertThrows { + (aliases.typeAliases as MutableMap>).remove("Person") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (aliases.typeAliases["Person"] as MutableSet).add("Sneaky") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (aliases.propertyAliases as MutableMap>>).remove("Person") + } + + @Suppress("UNCHECKED_CAST") + assertThrows { + (aliases.propertyAliases["Person"]!!["emailAddress"] as MutableSet).add("Sneaky") + } + } + + @Test + fun `equality is by content`() { + val one = SchemaAliases( + typeAliases = mapOf("Person" to setOf("Human")), + propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email"))), + ) + val two = SchemaAliases( + typeAliases = mapOf("Person" to setOf("Human")), + propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email"))), + ) + + assertEquals(one, two) + assertEquals(one.hashCode(), two.hashCode()) + assertTrue(one.toString().contains("Human")) + } +} diff --git a/dice/pom.xml b/dice/pom.xml index 09616eca..ba1c7432 100644 --- a/dice/pom.xml +++ b/dice/pom.xml @@ -61,7 +61,6 @@ org.jetbrains annotations - 26.0.1 dice @@ -37,9 +39,11 @@ flowchart TB ``` `dice` never depends on any other DICE module — it's the leaf of the graph, so every other module -can be added or removed without touching core logic. `dice-storage-autoconfigure` is the only -module that knows about Spring Boot autoconfiguration; plain `dice-storage` stays framework-neutral -so it can be wired by hand outside Spring Boot. +can be added or removed without touching core logic. `dice-metamodel` is a second leaf with no +edges: it stamps a schema, and depends only on Embabel's agent core types. +`dice-storage-autoconfigure` is the only module that knows about Spring +Boot autoconfiguration; plain `dice-storage` stays framework-neutral so it can be wired by hand +outside Spring Boot. ### Subsystem design docs @@ -62,6 +66,7 @@ Each subsystem below the module level has its own design note: - [durable-storage](durable-storage.md) — `dice-storage` backend, schema, indexes - [events](events.md) — `DiceEvent` model and emitters - [report](report.md) — `dice-report` rationale and structured reports +- [metamodel-versioning](metamodel-versioning.md) — `MetamodelVersion` stamping, per-type governance - [web-api](web-api.md) — REST surface (`DiscoveryController` and friends) ## System-level map diff --git a/docs/design/metamodel-versioning.md b/docs/design/metamodel-versioning.md new file mode 100644 index 00000000..b5ff7b86 --- /dev/null +++ b/docs/design/metamodel-versioning.md @@ -0,0 +1,375 @@ +# Metamodel versioning: stamping, governance, and history + +DICE stamps the governed part of a metamodel with a content hash, so a schema has an identity you +can record against extracted data and compare later. + +DICE extracts entities and relationships against a metamodel: the entity types, their labels and +properties, and the relationships allowed between them. That schema moves. It gets edited as a +domain is understood better, and it can diverge from what a live graph holds, because an +integration that used to declare a type can be switched off while its data stays behind. + +This note covers turning a mutable `DataDictionary` into an immutable stamp, deciding which types +the stamp is about, and keeping the stamps so history is answerable. Comparing two stamps, or a +stamp against a live graph, comes later — see [the tiers ahead](#the-tiers-ahead). + +The types live in `dice-metamodel`, a small pure-JVM module: `MetamodelVersion`, +`GovernedTypeSelector`, `DeclaredSchema`/`DeclaredSchemaSource`, `SchemaAliases`, and the +`MetamodelVersionStore` contract. It depends on Embabel's agent core types and nothing else. +`SchemaAliases` and the alias fields on `PropertySignature` and `MetamodelVersion` are +experimental; their shape may change before 1.0. + +## Declare, stamp, store + +```mermaid +flowchart LR + dict["DataDictionary
(mutable, whole domain)"] + sel["GovernedTypeSelector
which types we version"] + dss["DeclaredSchemaSource
(opt-in seam)"] + stamp["MetamodelVersion
contentHash"] + store[("MetamodelVersionStore")] + history["latestVersion / versionHistory
findVersion(schema, hash)"] + + dict --> dss + sel --> dss + dss -->|declare| stamp + stamp -->|saveVersion| store + store --> history +``` + +Three moving parts, in order. An application says what it governs (`DeclaredSchemaSource`). +Stamping freezes that into a `MetamodelVersion` with a content hash. The store keeps every stamp, +so the hash a piece of extracted data carries can be resolved back into the schema shape it stood +for. + +## Why a version is a content hash + +The live schema is mutable, so stamping means taking an immutable snapshot of it. +`MetamodelVersion.from(dataDictionary)` snapshots sorted entity type names, the full label set per +type, the full property *signature* set per type, and sorted relationship descriptors, then +fingerprints them as a SHA-256 `contentHash`. A proposition's schema attribution is answered through +the run that produced it (PRODUCED_BY_RUN); the run record carries the declared schema's content hash, +resolved by the extraction coordinator from the host's DeclaredSchemaSource. A per-proposition +denormalized copy is a coordinator concern for a later slice if reads demand it. + +Four choices in the fingerprint matter. + +**The schema name is excluded.** Two structurally identical schemas hash identically whatever +they're called, so a dev environment and a prod environment compare cleanly. The cost is that +`contentHash` alone can't tell two same-shaped schemas apart by name, which is why the store keys +on `(schemaName, contentHash)` together. + +**The hash is derived inside the class.** `MetamodelVersion` computes it from its own structural +fields; there is no constructor parameter for it. The hash is the store's natural key and what +`hasSameContentAs` compares. If a caller could supply it, two schemas with different types could +claim the same hash, compare equal, and overwrite each other in storage. + +**Properties go in as signatures, not names.** A property contributes its name, whether it holds a +value or points at another type, that value type (`string`, `integer`, ...) or target type name, +and its cardinality. With names alone, `age` turning from a string into an integer, or a single +`worksAt` becoming a list of them, would leave the hash untouched, though both change what the +graph can hold. Descriptions and semantic metadata are left out: they steer extraction, but they +don't change the shape of what gets stored, so re-wording one doesn't count as a schema change. + +**Every name, label, and signature component is length-prefixed** (`:`) before hashing, +and each set is preceded by its size. These names come from free-text and LLM extraction, and +routinely contain `;`, `[`, `=`, and spaces. A delimiter-joined encoding would let `["a;b"]` and +`["a", "b"]` produce the same digest, hiding a schema change that collapsed one into the other. +Length-prefixing makes the encoding unambiguous, so distinct content always yields a distinct hash. + +The hashed form is `types:|` followed, for each type name in sorted order, by the length-prefixed +name, an optional `typealiases:|` block, then `labels:|` and its sorted labels, then +`props:|` and its sorted signatures — each signature contributing name, kind, type, and +cardinality as four length-prefixed tokens, followed by an optional `aliases:|` block. Then +`rels:|` and the sorted relationship descriptors. The schema name appears nowhere. + +The two alias blocks are written only when they hold something, which is what lets aliases be added +to a shipped encoding at all. A schema that declares no former names renders exactly the bytes this +encoding produced before aliases existed, so every hash already recorded against it still resolves. +`MetamodelVersionTest` pins that. One test asserts the golden digest for the same dictionary stamped +four ways: `from(dictionary)`, `from(dictionary, GovernedTypeSelector.ALL)`, the same with an +explicit `SchemaAliases.NONE`, and the same with an explicitly empty +`SchemaAliases(emptyMap(), emptyMap())`. A second asserts it for a stamp rebuilt field by field +through the public constructor with an empty alias map, which is the path a storage mapper takes. + +The block shape is the one the rest of the encoding already uses: `:|` and then +length-prefixed entries in sorted order. Position keeps the two tags apart — the type block sits +between a type's name and its labels, the property block after a signature's fourth token — so +declaring `old` as a former type name and declaring it as a former property name are different +digests. + +Two things about the input. A `DataDictionary` can legally hold two domain types sharing a name but +differing in shape, so `from` unions their labels and properties per name. Keeping only the last +would drop a label from the fingerprint, and removing that label later wouldn't change the hash. +The same split can render one relationship descriptor twice, so the constructor sorts *and* +deduplicates the type and relationship lists: declaring a type once or splitting it in two is the +same schema, and has to be the same hash. + +The constructor is strict about the rest, too. It copies every collection into a JVM-immutable one, +down to the alias set inside each property signature, which arrives however the caller built it. +Kotlin's read-only view is a compile-time promise that a Java caller sees straight through, so it +wouldn't stop anything being reshaped out from under the precomputed hash. The constructor also +rejects a label or property map keyed by a type missing from `entityTypeNames`: only listed types +are walked when hashing, so such an entry would never reach `contentHash`, and two structurally +different stamps could end up sharing the store's natural key. `MetamodelVersion` is a plain class +rather than a `data class` for the same reason: a generated `copy()` would hand its arguments +straight to the fields and skip all of that. + +The encoding is a persisted format. `MetamodelVersionTest` pins the digest of a fixed schema to a +literal, because changing how the fingerprint is built orphans every hash already recorded against +it, and needs a migration. + +## Versioning is per type, and opt-in + +A DICE domain is rarely all one thing. Part of it is closed-world: the types you have committed to, +whose shape you want to notice changing. The rest is open-world: exploratory types that extraction +proposes, that come and go, and that nobody has decided about yet. + +Stamping the whole dictionary treats both alike. Every new exploratory type then produces a new +content hash, the version history fills with entries nobody chose, and comparisons stop meaning +anything. Governance is therefore per type. `GovernedTypeSelector` is a predicate over +`DomainType`, and `MetamodelVersion.from(dictionary, selector)` stamps only what it governs: + +```kotlin +val governed = setOf("Person", "Company") +val version = MetamodelVersion.from(dataDictionary, GovernedTypeSelector { it.name in governed }) +``` + +Add, remove or reshape an ungoverned type and `contentHash` doesn't move. Do the same to a governed +one and it does. `from(dictionary)` with no selector governs everything, which is the right stamp +for a domain that is closed-world throughout. + +The model is Hibernate's `@Version`: governance is declared per entity, and nothing is versioned by +default. + +Relationships follow the type that declares them. A governed type's outgoing relationship is part of +that type's declared shape, so it stays in the stamp even when it points at an ungoverned type; a +relationship declared *by* an ungoverned type is left out entirely. Without that rule, an +exploratory type wiring itself to `Person` would move `Person`'s hash. + +`DeclaredSchema.from(dictionary, selector)` applies the same rule to both halves of a declaration, +which is why it exists. A declaration carries the bare relationship type names alongside the stamp, +because `relationshipNames` holds rendered `From-[name]->To` descriptors and reverse-parsing one is +ambiguous when the names themselves can contain a `-[...]->`-shaped substring. Building the stamp +from a governed subset while taking the names from the whole dictionary would declare relationships +the stamp never covered, and the mismatch would only surface much later, as phantom disagreement. + +## The opt-in seam + +`DeclaredSchemaSource` is where an application states what it governs: a single method returning a +`DeclaredSchema`. It has no default implementation, because there is no default declared schema. A +consuming app maps whatever it already uses to define its types: + +```kotlin +class MyAppDeclaredSchemaSource( + private val dataDictionary: DataDictionary, +) : DeclaredSchemaSource { + + private val governed = GovernedTypeSelector { it.name in setOf("Person", "Company") } + + override fun declare(): DeclaredSchema = DeclaredSchema.from(dataDictionary, governed) +} +``` + +Versioning starts here: with no declared schema, nothing is stamped. The Spring wiring that lands +in a later slice activates only when a `DeclaredSchemaSource` bean is present, so an application +that hasn't decided what it governs is left alone. + +## Declared renames + +Renaming a type or a property is the change a content hash reads worst. `email` becoming +`emailAddress` is one property with a new spelling, and a stamp comparison sees a removal and an +addition, which is the same shape as deleting a property and inventing an unrelated one. A declared +alias says what the name used to be, so the comparison can pair the two. + +Iceberg and Delta hold identity in stable field ids assigned when a column is created, leaving the +name as a label over an identity the format already carries. DICE has no id to assign. Its types +come from LLM extraction against a `DataDictionary` an application edits, and the upstream +`PropertyDefinition` has nowhere to put an id even if DICE minted one. The adapted form is the +former name itself, declared at the moment the rename is made: + +```kotlin +DeclaredSchema.from( + dataDictionary, + governed, + SchemaAliases( + typeAliases = mapOf("Organisation" to setOf("Company")), + propertyAliases = mapOf("Person" to mapOf("emailAddress" to setOf("email"))), + ), +) +``` + +Types get the mechanism as well as properties because a type rename is the more destructive of the +two: every proposition labelled with the old name and every referrer pointing at it move at once. + +Alias names are exact and case-sensitive. LLM extraction drifts on case, and folding `worksAt` into +`worksat` here would pair two names nobody declared as the same one. + +Aliases accumulate. A type renamed `A` to `B` to `C` declares `{A, B}`, so a comparison across +non-adjacent stamps still pairs. Retiring a name means deleting it from the declaration. + +`SchemaAliases` is a declaration-time input; the stamp carries the result. `MetamodelVersion.from` +decorates each signature with the former names declared for its property, inside the per-type loop +right after the signature is read, because the stamp is immutable and hashes at construction. Type +aliases land in `entityTypeAliases`. Aliases declared for a type the selector doesn't govern are +dropped, along with everything else about an ungoverned type. + +A property alias equal to the property's own name, or naming a property that still exists on both +sides of a comparison, matches nothing: nothing looks data up by property name, so a stale property +alias can mislead nothing. It stays part of the signature and moves the hash. A type alias naming +another declared type is a different case, and the third guard below rejects it. + +Comparing two stamps is the next slice. Until it lands, an alias is a recorded intention that moves +the hash and nothing more. + +### The four guards + +```mermaid +flowchart TB + aliases["SchemaAliases
typeAliases, propertyAliases"] + seam["MetamodelVersion.from / DeclaredSchema.from"] + ctor["MetamodelVersion constructor
(also the storage reconstruction path)"] + g1{"alias keys all in
entityTypeNames?"} + g2{"every alias set
non-empty?"} + g3{"alias names another
declared type?"} + g4{"aliases on a property name
with two signatures?"} + stamp["Stamp: entityTypeAliases and
PropertySignature.aliases, both hashed"] + reject["IllegalArgumentException
naming the alias to retire"] + + aliases --> seam + seam -->|"ungoverned keys dropped,
signatures decorated"| ctor + ctor --> g1 + g1 -->|no| reject + g1 -->|yes| g2 + g2 -->|no| reject + g2 -->|yes| g3 + g3 -->|yes| reject + g3 -->|no| g4 + g4 -->|yes| reject + g4 -->|no| stamp +``` + +**Alias-map keys are a subset of `entityTypeNames`.** Only listed types are walked when hashing, so +an entry keyed by anything else never reaches `contentHash`. This is the rule the label and property +maps already follow. + +**Every alias set is non-empty.** An entry mapping a type to no former names hashes differently from +having no entry at all while saying the same thing, so two stamps of one schema could land on two +different natural keys. `SchemaAliases` drops empty sets on the way in, which keeps a harmless +declaration from becoming an error at the seam. + +**No declared type name appears in another type's alias set.** This is the reuse collision. A schema +renames `Company` to `Organisation`, keeps `{Company}` as the alias, and later declares a fresh +`Company` for something unrelated. A live type now shares a retired name, and the comparison has two +bad options: sweep the new `Company`'s data into the renamed type's quarantine matching, or treat +the old label as declared forever and never report drift on it. Reusing a retired name therefore +requires deleting the alias that still claims it first. A type listing its own name is fine — a +rename to `B` and back to `A` accumulates `{A, B}` — because the guard is about other types' names. + +**No aliases on a property name a type holds more than one signature for.** Two same-named domain +types can each declare `age` with a different shape, and the union keeps both signatures. An old +name has no way to say which of the two it meant, so the declaration is refused. + +All four run in the `MetamodelVersion` constructor, so the public constructors and the storage +mapper's reconstruction path are covered. The last two also run at `MetamodelVersion.from` and +`DeclaredSchema.from`, which is where duplicates and reused names actually become visible, and where +the message can name the alias to retire. + +### What the duplicate-name refusal costs + +A dictionary can legally evolve into duplicate-hood. Someone adds a second `Person` declaration +carrying its own `age`, and a name that had one signature now has two. If an alias was standing on +that name, every stamp and every drift check from that moment throws, with a message naming the +alias to retire. That is a loud, conservative hard stop on a schema that was fine the day before. + +Retiring the alias clears the throw and costs the pairing. The name then diffs through the duplicate +fallback as a removal plus an addition, the drift policy reads that as lossy, and an additive +evolution earns a quarantine sweep. Both outcomes cost more than an ordinary signature change would, +and both beat the only other option, which is guessing which of the two signatures the old name +referred to. + +## Stamp provenance waits for a caller + +A stamp says nothing about who or what caused it. That is deliberate for now. Recording the cause +means fixing a type for it, deciding how long its strings may be, finding it somewhere to live in +every backend, and settling what a re-save does to a value that is already there — a pile of +commitments made on behalf of a caller that doesn't exist yet. Nothing in DICE stamps with a cause +today: `MetamodelVersion.from` builds a stamp from a dictionary, and the drift check that arrives +in a later slice re-stamps the +declared schema without anything to attribute it to. + +Provenance returns with the first stamping caller that records it. Whoever that caller is will +settle the shape, which is a better basis for the decision than a guess made here. + +Snowflake's `SCHEMA_EVOLUTION_RECORD` shows where it lands when it does arrive: the evolution event +sits alongside the schema rather than inside the schema's identity. Two stamps of one schema taken +for different reasons are the same schema, and the hash is the store's natural key, so hashing the +cause would give one schema as many identities as it had causes. + +## History accumulates + +`MetamodelVersionStore` is a port with four operations: `saveVersion`, `latestVersion`, +`versionHistory`, and `findVersion(schemaName, contentHash)`. There is no delete. Correlating what +a graph holds today against the stamp it was extracted under only works while the old stamps are +still there. + +`MetamodelVersion` and `MetamodelVersionStore` carry no context dimension — one declared schema +serves every tenant, the default where a `DataDictionary` is application-wide. Drift reports and +observation are per `ContextId`; a reader arriving from the drift store should not assume versions +are scoped. + +`saveVersion` is an upsert on `(schemaName, contentHash)`. Re-saving is idempotent, because the key +carries the content: the hash is derived from exactly the fields a re-save would overwrite, so +anything landing on an existing key has identical content by construction. The interface doesn't +promise append-only storage, and an implementation isn't expected to reject a re-save. + +This is where schema registries have already landed. AWS Glue Schema Registry and Confluent Schema +Registry both identify a schema version by a fingerprint of its content and answer a re-registration +of an identical schema with the existing version rather than a new one. Deriving identity from the +content is what makes registration idempotent, and it is why a client that re-registers on every +boot doesn't inflate the history. DICE keys on `(schemaName, contentHash)` for that reason, and an +application stamping its declared schema on every start is exactly the client those registries are +built for. + +`findVersion` resolves a recorded hash back to the schema shape it named. The default scans +`versionHistory`, which is correct for any implementation but reads the whole history to answer a +keyed question; a backend that can push the lookup down to the database should override it. This +module ships no implementation. Storage is a separate concern, and a stamp is useful in memory +before anything durable exists. + +## Plain classes, not data classes + +`MetamodelVersion`, `DeclaredSchema` and `SchemaAliases` each write their own `equals`, `hashCode` +and `toString`. That is one deliberate pattern, for one reason: each of them copies what the +constructor is handed into a JVM-immutable collection in its body, and a `data class` cannot do +that. A constructor `val` takes no initialiser, so the generated `equals` and `copy` would read the +raw arguments and skip the copy, and a stamp whose collections could still be changed from the +outside would disagree with its own precomputed hash. + +`PropertySignature` is the exception that proves it. It is a `data class`, and its `aliases` set is +therefore held as handed in; `MetamodelVersion` copies that set into an immutable one when it takes +a signature. The KDoc on each class says as much, and this section is here so the pattern is read +as a module decision and not raised class by class. + +Two notions of equality live on `MetamodelVersion`, and both are meant. `equals` compares the schema +name along with the content, so a `Set` keys the way the store does. `contentHash` +and `hasSameContentAs` leave the name out, so two schemas with the same shape under different names +compare equal there. That is also why the store's natural key is `(schemaName, contentHash)` and not +the hash alone: history is per schema, and one schema adopting a shape another had earlier must not +land on the other schema's record. + +## The tiers ahead + +Versioning is the first of three escalating tiers, shipped in that order. + +**Stamp and observe** is this slice: identity and history. + +**Detect and report** comes next: comparing two declared stamps, comparing a declaration against +what a live graph contains, and recording the result. + +**Quarantine** is last: acting on a lossy change by marking affected propositions stale rather than +deleting them. + +Each tier builds on the one below it, and each is useful on its own. You can stamp for a year +without detecting, and detect for a year without quarantining. Rejecting undeclared types at write +time is not on the list yet: extraction is LLM-driven, a type nobody declared is often a real +finding, and discarding it is the one thing that can't be undone later. diff --git a/pom.xml b/pom.xml index 4bf74b0e..988fb0fc 100644 --- a/pom.xml +++ b/pom.xml @@ -28,6 +28,7 @@ dice-storage-autoconfigure dice-report dice-ingestion + dice-metamodel dice-integration-tests dice-user-guide @@ -48,6 +49,11 @@ and reference it from the surefire system properties below. --> 1.41 + + 26.0.2 @@ -81,6 +87,11 @@ dice-report ${project.version} + + com.embabel.dice + dice-metamodel + ${project.version} + @@ -88,6 +99,12 @@ drivine4j-spring-boot-starter ${drivine.version} + + + org.jetbrains + annotations + ${jetbrains.annotations.version} +