feat(dice-storage): Drivine drift-report store and observed-schema source

dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/MetamodelDiff.kt

@@ -611,12 +611,18 @@ class MetamodelDiff(
  * about whether a declared property's shape matches what the graph stores. Property signatures are
  * compared where both sides have them: declared against declared, in [MetamodelDiff].
  *
- * **A declared label counts as declared.** A graph reports labels, and a type usually carries more
- * than one: declaring `Person` with parent `Agent` puts both labels on every `Person` node. So the
- * declared side of the drift comparison is every entity type name *plus* every label those types
- * declare. Without the labels, an inherited label would be reported as undeclared drift on a schema
- * nobody had changed. [unobservedEntityTypes] stays on the type names alone, because "declared but
- * with no data" is a statement about types, and a parent label was never a type in its own right.
+ * **What counts as declared depends on what kind of name was observed**, per
+ * [ObservedSchema.EntityTypeBasis]. A graph reports labels, and a type usually carries more than
+ * one: declaring `Person` with parent `Agent` puts both labels on every `Person` node, so against a
+ * [ObservedSchema.EntityTypeBasis.GRAPH_LABELS] observation the declared side is every entity type
+ * name *plus* every label those types declare — without the labels, an inherited label would be
+ * reported as undeclared drift on a schema nobody had changed. A mention's `type` is domain data an
+ * extractor wrote, living in its own namespace apart from graph labels, so against a [ObservedSchema.EntityTypeBasis.MENTION_TYPES]
+ * observation the declared side stays on entity type names, with no widening to inherited labels: a
+ * mention typed `Agent` conforms only when `Agent` is itself a declared type, whatever labels a
+ * governed `Person` carries. [unobservedEntityTypes] stays on the type names alone either way,
+ * because "declared but with no data" is a statement about types, and a parent label was never a
+ * type in its own right.
  *
  * @property declared The schema as declared at snapshot time, stamp and bare relationship names.
  * @property observedSchema What the live graph held at snapshot time.

dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/ObservedSchema.kt

@@ -36,17 +36,62 @@ import java.util.Objects
  * contributing types that come and go, so what is in the graph can drift from what was declared,
  * and this module needs a way to talk about that without depending on a particular graph driver.
  *
- * @property entityTypeNames Entity type (label) names observed in the graph.
+ * A snapshot can carry both kinds of entity name at once, in two sets. A whole-graph observation
+ * reads labels off the database catalogue and mention types off the stored data, and those two
+ * questions have different declared sides, so keeping them apart is what lets each be judged by its
+ * own rule. [entityTypeBasis] says which kind [entityTypeNames] holds; [mentionTypeNames] is always
+ * mention types and needs no tag.
+ *
+ * @property entityTypeNames Entity type names observed. What counts as a name depends on
+ *   [entityTypeBasis]: a Neo4j graph label, or a mention's own `type` field.
  * @property relationshipTypeNames Relationship type names observed in the graph.
  * @property capturedAt When this snapshot was taken.
+ * @property entityTypeBasis What kind of name [entityTypeNames] holds, which decides what the
+ *   declared side of a comparison has to be. See [EntityTypeBasis].
+ * @property mentionTypeNames Types the stored data claims for its entity mentions, held apart from
+ *   [entityTypeNames] so a comparison can judge them by the mention rule whatever
+ *   [entityTypeBasis] says. Empty when the observation had no way to ask the data, which is the
+ *   case for any source that only reads a label catalogue.
  */
 @ApiStatus.Experimental
-class ObservedSchema(
+class ObservedSchema @JvmOverloads constructor(
     entityTypeNames: Set<String>,
     relationshipTypeNames: Set<String>,
     val capturedAt: Instant,
+    val entityTypeBasis: EntityTypeBasis = EntityTypeBasis.GRAPH_LABELS,
+    mentionTypeNames: Set<String> = emptySet(),
 ) {
 
+    /**
+     * What kind of name [ObservedSchema.entityTypeNames] holds. A [DeclaredObservedDiffer] needs to
+     * know this, because the two kinds of name compare against different declared sets.
+     */
+    enum class EntityTypeBasis {
+
+        /**
+         * A Neo4j label, read off `db.labels()` on a whole-graph observation. A node carries every
+         * label in its declared type's hierarchy, so declaring `Person` with parent `Agent` puts both
+         * labels on every `Person` node. The declared side of a comparison against this basis has to
+         * include every label the declaration's types carry, on top of the type names themselves, or
+         * an inherited parent label reads as undeclared drift on a schema nobody touched.
+         */
+        GRAPH_LABELS,
+
+        /**
+         * The `type` a mention was extracted as, read off `Mention.type` on a context-scoped
+         * observation. An observation that carries labels in [ObservedSchema.entityTypeNames] puts
+         * its mention types in [ObservedSchema.mentionTypeNames], where this same rule applies to
+         * them. This is domain data an extractor wrote, living in its own namespace apart from graph labels, and a graph's
+         * label hierarchy has no bearing on it: a mention typed `Agent` claimed to be an `Agent`, and
+         * that claim stands or falls on whether `Agent` is itself a declared type, whatever labels a
+         * governed `Person` happens to carry. The declared side of a comparison against this basis
+         * stays on declared type names (plus their declared former names — old data can still carry a
+         * type's pre-rename spelling); it must not widen to include inherited labels, or an undeclared
+         * mention type escapes detection by riding a governed type's parent label.
+         */
+        MENTION_TYPES,
+    }
+
     // Both sets are copied into immutable ones, keeping the order they arrived in. A snapshot
     // describes one moment and must not change afterwards, and a backend typically builds these from
     // a mutable set it fills as it walks query results. Kotlin's read-only `Set` is a compile-time
@@ -57,17 +102,22 @@ class ObservedSchema(
 
     val relationshipTypeNames: Set<String> = immutableCopy(relationshipTypeNames)
 
+    val mentionTypeNames: Set<String> = immutableCopy(mentionTypeNames)
+
     override fun equals(other: Any?): Boolean =
         other is ObservedSchema &&
             entityTypeNames == other.entityTypeNames &&
             relationshipTypeNames == other.relationshipTypeNames &&
-            capturedAt == other.capturedAt
+            capturedAt == other.capturedAt &&
+            entityTypeBasis == other.entityTypeBasis &&
+            mentionTypeNames == other.mentionTypeNames
 
-    override fun hashCode(): Int = Objects.hash(entityTypeNames, relationshipTypeNames, capturedAt)
+    override fun hashCode(): Int =
+        Objects.hash(entityTypeNames, relationshipTypeNames, capturedAt, entityTypeBasis, mentionTypeNames)
 
     override fun toString(): String =
         "ObservedSchema(entityTypeNames=$entityTypeNames, relationshipTypeNames=$relationshipTypeNames, " +
-            "capturedAt=$capturedAt)"
+            "capturedAt=$capturedAt, entityTypeBasis=$entityTypeBasis, mentionTypeNames=$mentionTypeNames)"
 
     private companion object {
 

dice-metamodel/src/main/kotlin/com/embabel/dice/metamodel/support/StructuralMetamodelDiffer.kt

@@ -33,6 +33,10 @@ import com.embabel.dice.metamodel.PropertySignature
  * content hash is built from, which is what makes an empty diff and an equal hash mean the same
  * thing.
  *
+ * An observation can carry two kinds of entity name, and [diffAgainstObserved] judges each by its
+ * own rule: graph labels against the declared side its basis calls for, mention types against
+ * declared type names and their declared former names alone. See [ObservedSchema.mentionTypeNames].
+ *
  * Two rules it keeps throughout. Sets are compared as sets, never as a delimiter-joined projection,
  * because a label or property name can contain a comma or a space, which is routine when names come
  * from LLM extraction, and joining would collapse two different sets into a false "unchanged".
@@ -138,35 +142,61 @@ class StructuralMetamodelDiffer : MetamodelDiffer, DeclaredObservedDiffer {
         val declaredTypes = declared.version.entityTypeNames.toSet()
         val observedTypes = observed.entityTypeNames
 
-        // What a graph reports is labels, and a type carries every label in its hierarchy: declare
-        // `Person` with parent `Agent` and every Person node comes back carrying both. Comparing
-        // observed labels against type names alone would call `Agent` undeclared drift on a schema
-        // nobody had touched, so the declared side of the drift check is the type names plus every
-        // label those types declare.
+        // Declared former names count as declared on both bases below. Nodes written before a type
+        // was renamed keep the old label, and a mention extracted before the rename keeps the old
+        // type name, and either way the rename was declared, so the old name is known. Leaving it out
+        // would report a declared rename as drift on every check from then on.
         //
         // A declared name can also be fully qualified where the observed one is simple. The stamp
         // holds `com.example.Person` for a JVM-backed type, extraction records the mention as
-        // `Person`, and the graph reports `Person`. Both spellings of every declared type go on the
-        // declared side, through DeclaredSchema.entityTypeOwnLabels.
-        //
-        // Declared former names count as declared too. Nodes written before a type was renamed keep
-        // the old label, and the rename was declared, so the old label is known. Leaving it out
-        // would report a declared rename as drift on every check from then on. A former name is a
-        // declared name, so it brings its own label with it the same way.
+        // `Person`, and the graph reports `Person`. Both spellings of every declared type — and of
+        // every declared former name — go on the declared side, through
+        // DeclaredSchema.entityTypeOwnLabels and ownLabelsOf.
         val declaredAliases = declared.version.entityTypeAliases.values.flatten()
-        val declaredLabels = declaredTypes +
+        val declaredEitherSpelling = declaredTypes +
             declared.entityTypeOwnLabels +
-            declared.version.entityTypeLabels.values.flatten() +
             declaredAliases +
             ownLabelsOf(declaredAliases)
 
+        // What counts as "declared" on the observed side depends on what kind of name is being
+        // compared; see ObservedSchema.EntityTypeBasis.
+        val declaredEntityTypeBasis = when (observed.entityTypeBasis) {
+            // A graph reports labels, and a type carries every label in its hierarchy: declare
+            // `Person` with parent `Agent` and every Person node comes back carrying both. Comparing
+            // observed labels against type names alone would call `Agent` undeclared drift on a
+            // schema nobody had touched, so the declared side of the drift check is the type names
+            // plus every label those types declare.
+            ObservedSchema.EntityTypeBasis.GRAPH_LABELS ->
+                declaredEitherSpelling + declared.version.entityTypeLabels.values.flatten()
+            // A mention's `type` is domain data an extractor wrote, living in its own namespace
+            // apart from graph labels, so a
+            // governed type's inherited parent label has no bearing on it: a mention typed `Agent`
+            // only conforms when `Agent` is itself a declared type or former name, whatever labels a
+            // governed `Person` carries. Widening this to declared labels would let an undeclared
+            // mention type escape detection by riding a governed type's parent label.
+            ObservedSchema.EntityTypeBasis.MENTION_TYPES ->
+                declaredEitherSpelling
+        }
+
         // A type the host's dictionary names but the selector leaves outside governance is a known
         // type, and the drift check has to recognise it as such. It gets its own excluded set,
-        // separate from declaredLabels, because it must stay out of unobservedEntityTypes below: a
+        // separate from the declared basis, because it must stay out of unobservedEntityTypes below: a
         // governance-exempt type with no data isn't the informational case that bucket describes.
         // These names come off the same dictionary the governed ones do, so they can be fully
         // qualified in the same way, and their own labels are excluded alongside them.
-        val excludedFromDrift = declaredLabels +
+        val excludedFromDrift = declaredEntityTypeBasis +
+            declared.ungovernedEntityTypeNames +
+            ownLabelsOf(declared.ungovernedEntityTypeNames)
+
+        // Mention types arrive in their own set, and they are judged by the mention rule whatever
+        // basis the names above carry. A whole-graph observation reports both kinds at once — labels
+        // off the database catalogue, mention types off the stored propositions — and one basis tag
+        // can only describe one of them. Judging the pair together under GRAPH_LABELS would widen the
+        // declared side to inherited labels for the mention half too, which is the escape hatch
+        // MENTION_TYPES exists to close: a mention typed `Agent` would pass under a schema that
+        // governs `Person` with parent label `Agent` and declares no `Agent` type at all.
+        val observedMentionTypes = observed.mentionTypeNames
+        val mentionTypesExcludedFromDrift = declaredEitherSpelling +
             declared.ungovernedEntityTypeNames +
             ownLabelsOf(declared.ungovernedEntityTypeNames)
 
@@ -193,11 +223,19 @@ class StructuralMetamodelDiffer : MetamodelDiffer, DeclaredObservedDiffer {
         return DeclaredObservedDiff(
             declared = declared,
             observedSchema = observed,
-            driftedEntityTypes = canonical(observedTypes - excludedFromDrift),
+            driftedEntityTypes = canonical(
+                (observedTypes - excludedFromDrift) + (observedMentionTypes - mentionTypesExcludedFromDrift),
+            ),
             driftedRelationshipTypes = canonical(observedRels - relsExcludedFromDrift),
+            // Data mentioning a declared type is that type being observed, the same as a graph label
+            // reporting it, so both sets answer this bucket.
             unobservedEntityTypes = canonical(
                 declaredTypes.filterNot {
-                    isObserved(it, declared.version.entityTypeAliases[it].orEmpty(), observedTypes)
+                    isObserved(
+                        it,
+                        declared.version.entityTypeAliases[it].orEmpty(),
+                        observedTypes + observedMentionTypes,
+                    )
                 },
             ),
             unobservedRelationshipTypes = canonical(declaredRels - observedRels),