Skip to content

feat(dice-storage): Drivine drift-report store and observed-schema source - #87

Merged
jimador merged 10 commits into
mainfrom
feat/metamodel-drift-store
Sep 9, 2026
Merged

jimador merged 10 commits into
mainfrom
feat/metamodel-drift-store

Conversation

@jimador

@jimador jimador commented Aug 30, 2026 •

Copy link
Copy Markdown
Collaborator

PR 5 of the metamodel train, stacked on #86. The Drivine half of the drift story. DrivineDriftReportStore runs three distinct Cypher reads (all / global / one context), each scoping before limiting, newest-first with a persisted per-schema sequence as the deterministic tie-break, and injective scope keys so a context id equal to the old sentinel cannot overwrite a global report. DrivineObservedSchemaSource observes the whole graph via label and relationship-type queries plus a mention-type query, and observes a context via the proposition walk. DrivineMetamodelVersionStore declares SweptBaselineStore, binding its markSwept/sweptVersion semantics. A full-loop integration test runs declare → ingest an undeclared mention → check → report persisted and hash-resolvable → sweep → proposition quarantined, against a real Neo4j container.

Changed in this review round:

  • The whole observation takes one read snapshot — one transaction, read-committed, stated as such in the KDoc, closing the window where concurrent writes produced a combined observation that never existed at any instant.
  • Mention types are separated from graph labels. ObservedSchema.EntityTypeBasis splits the declared set: the mention-type arm compares against declared type names plus aliases, so a mention typed with a governed type's inherited parent label no longer passes governance silently. Global mention-type observation is new too — a recorded mention type that projected no node was previously invisible to unscoped checks.
  • Graph ownership is derived from the registered schemas. DiceStorageSchema is the contract each Cypher-backed store's schema object implements; DiceOwnedSchema.of(registered) derives the bookkeeping exclusion from that bean list, and the DDL comes off the same list via diceStorageCatalog, so a store cannot get its constraints while sitting outside the exclusion. The hand-written label map is gone; a store registered nowhere stays visible to observation, which is what a drift check should report.
  • The label side counts only labels some node wears — a uniqueness constraint on an empty graph no longer reads as permanent drift — and Drivine's own _DrivineSchema inventory label is excluded by name.
  • Activating @Transactional exposed that Boot 4.1 moved transaction autoconfiguration into spring-boot-transaction; the module now depends on it, and the dedup boundary runs REQUIRES_NEW.

Breaking changes: none. New stores and one new constraint set for hosts adopting drift persistence. Behavioral notes: DrivineObservedSchemaSource gains a required constructor argument (branch-new type), and the transaction activation is a genuine runtime change on upgrade, stated in the CHANGELOG.

Status: the derived-ownership surface (DiceStorageSchema, diceStorageCatalog, DiceOwnedSchema) is EXPERIMENTAL; a store takes part by being registered.

sequenceDiagram
    participant R as DriftCheckRunner
    participant S as DrivineObservedSchemaSource
    participant N as Neo4j
    R->>S: observe(contextId)
    activate S
    Note over S,N: one transaction, one read snapshot
    S->>N: labels with nodes, relationship types
    S->>N: mention types
    N-->>S: combined observation
    deactivate S
    S-->>R: ObservedSchema with basis
    Note over R: MENTION_TYPES compares against declared names plus aliases
    Note over R: GRAPH_LABELS compares against the full label closure
    Note over R: registered DiceStorageSchema beans define the exclusion
Loading

Next in stack: #88 — opt-in Spring Boot wiring and the operator surface.

  • Release takes the quarantine keys off the node. The Drivine property bag is merge-only, so a release that dropped previousStatus and reason from the proposition's metadata left both on the node. DrivinePropositionRepository now removes every metadata.* property the saved metadata no longer names, with one dynamic REMOVE after the node write (Neo4j 5.24+). The drift integration test checks both keys are gone after a release.

@jimador
jimador force-pushed the feat/metamodel-drift-store branch from 7ca29f9 to 39b3afd Compare August 31, 2026 01:16
@jimador
jimador force-pushed the feat/metamodel-drift-store branch from 39b3afd to fc4b567 Compare August 31, 2026 04:54
@jimador
jimador force-pushed the feat/metamodel-drift-store branch from fc4b567 to da61530 Compare August 31, 2026 05:23
@jimador
jimador force-pushed the feat/metamodel-drift-store branch 2 times, most recently from 83d1230 to 9c18815 Compare August 31, 2026 06:35
@jimador
jimador force-pushed the feat/metamodel-drift-store branch from 9c18815 to 4968354 Compare August 31, 2026 18:46
@jimador
jimador force-pushed the feat/metamodel-drift-store branch from 4968354 to 55dce72 Compare August 31, 2026 19:40
@jimador
jimador force-pushed the feat/metamodel-drift-store branch from 55dce72 to 33d243a Compare September 1, 2026 03:41
@jimador
jimador marked this pull request as ready for review September 1, 2026 04:07
@jimador
jimador force-pushed the feat/metamodel-drift-store branch from 33d243a to 4bf7e86 Compare September 1, 2026 12:25
@jimador
jimador force-pushed the feat/metamodel-drift-store branch from 4bf7e86 to 5aa6772 Compare September 2, 2026 10:51
@jimador
jimador force-pushed the feat/metamodel-drift-store branch from 5aa6772 to ac087aa Compare September 2, 2026 14:11
@jimador
jimador requested a review from igordayen September 2, 2026 20:41
@jimador
jimador force-pushed the feat/metamodel-drift-store branch from 799d5a2 to c17d51a Compare September 2, 2026 20:52

@igordayen igordayen left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jimador - few comments from my side and codex. thank you

Comment thread dice-storage/pom.xml
@jimador
jimador force-pushed the feat/metamodel-drift-store branch from 2482222 to 6f1db2d Compare September 7, 2026 03:35
@igordayen

Copy link
Copy Markdown

@jimador - appears rebase is now required

@jimador
jimador requested a review from igordayen September 8, 2026 18:44
@jimador

jimador commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator Author

@igordayen both threads are answered above and the annotations version is fixed in 6f1db2d. Could you take another look? This is the last unapproved PR in the metamodel stack.

@igordayen

Copy link
Copy Markdown

@igordayen both threads are answered above and the annotations version is fixed in 6f1db2d. Could you take another look? This is the last unapproved PR in the metamodel stack.

@jimador yes, i wanted to approve, but it appears rebase is still required.

@jimador
jimador force-pushed the feat/metamodel-drift-store branch from 6f1db2d to 6177b7a Compare September 8, 2026 20:30
@jimador

jimador commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator Author

@igordayen rebased. The whole metamodel stack (#83 through #88 and #106) is now on top of main after the user-guide merge, replayed commit for commit with no conflicts; the only difference between the old and new tops is main's own change. Full clean install at the top of the stack is green. Ready for your approval whenever you get a moment.

@jimador

jimador commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator Author

Heads up on the red checks across this stack: every run dies in the first seconds with 401 Unauthorized from repo.embabel.com/artifactory/libs-snapshot while resolving embabel-agent-dependencies:1.5.0-SNAPSHOT. The Build workflow reads that repository anonymously and anonymous reads are returning 401 right now (main's own Deploy Snapshots run at 20:14 UTC failed the same way). The extraction train went green minutes earlier only because its Maven cache already held the BOM. Nothing in the rebase; a rerun once the repository is back should clear it.

@jimador
jimador force-pushed the feat/metamodel-drift-store branch from 6177b7a to eaef7b9 Compare September 8, 2026 20:47
Base automatically changed from feat/metamodel-drift to main September 8, 2026 20:59
@jimador
jimador force-pushed the feat/metamodel-drift-store branch from eaef7b9 to 38e07f2 Compare September 8, 2026 20:59
…urce

Three scoped drift-report reads that scope before limiting, ordered by
capture instant with a persisted per-schema sequence tie-break so pages
are repeatable; injective scope keys (global vs ctx:<id>). Observed
schema identifies dice bookkeeping by property shape rather than name,
so same-named domain types stay visible and the metamodel never observes
its own nodes as drift. End-to-end drift-check IT: declare, ingest an
undeclared mention, live run, report persisted, proposition quarantined.

Refs #45; stacks on feat/metamodel-drift.

Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
Comment and doc text only; no code change.

Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
…labels

The whole observation now comes from a single read transaction, so a
concurrent write landing between two schema queries can no longer produce
a combined observation the graph never held. Neo4j reads committed data
per row, so the KDoc states what that leaves open rather than claiming
snapshot isolation.

ObservedSchema gains an entityTypeBasis. A mention's type is domain data
an extractor wrote, living in its own namespace apart from graph labels,
so a MENTION_TYPES observation compares against declared type names and
aliases with no widening to inherited labels. Before this, a mention
typed Agent conformed whenever Agent was a parent label of a governed
Person, and an undeclared type rode into the graph on that. GRAPH_LABELS
keeps the previous semantics and stays the default, and @jvmoverloads
preserves the old three-argument constructor for Java callers.

DrivineMetamodelVersionStore now overrides sweptVersion and markSwept,
tracking sweptContentHash on the schema's counter node, so the durable
store follows the baseline independently of write order.

Enabling transaction management is the larger change here. dice-storage's
test harness never enabled it, and neither did production: Spring Boot
4.1 moved TransactionAutoConfiguration into spring-boot-transaction,
which nothing on the classpath brought in. Every @transactional in the
module was decorative. Adding the module dependency makes them real, and
Boot's configuration backs off for a consumer who already enables it.

That activation exposed a latent assumption. DrivinePropositionRepository
documents holding its stripe lock across the commit, which only held
while its TransactionTemplate ran its own transaction; an ambient
transaction made it join instead, moving the commit outside the lock.
REQUIRES_NEW restores it, pinned by a test that pauses the writer inside
beforeCommit and admits a sibling only there.

Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
The unscoped observation read graph labels only, so an undeclared
mention type with no same-named projected label never reached the
default drift check, and a mention typed as a governed type's inherited
label passed by riding it. ObservedSchema now carries mention types as
their own set: the differ judges labels under the observation's basis
and mention types under the mention rule, unioning the drift, and the
Drivine source fills both from one snapshot. The bookkeeping exclusion
derives from the storage schema definitions, so a host domain node
shaped like DICE bookkeeping stays observed. DrivineMetamodelVersionStore
declares SweptBaselineStore, binding its existing swept tracking to the
contract that moved there.

Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
Drivine writes _DrivineSchema for its own schema bookkeeping. DICE's
schema definitions cannot derive a label the library owns, so every
whole-graph check reported it as drift. Infrastructure labels get their
own enumerated exclusion beside the derived catalog, matched by exact
name, with a test proving an unknown label still drifts. Also retires
the word seam from the module docs and CHANGELOG.

Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
The bookkeeping exclusion was a hand list of four schema objects, and
the whole-graph observation read db.labels(), which Neo4j populates for
a label the moment a constraint names it, nodes or none. Any store a
later slice adds arrived as permanent whole-graph drift: its labels
were in the graph and the list had never heard of them. Ownership now
derives from the registered DiceStorageSchema beans, so DDL and
exclusion come off one list and a schema that registers is a schema the
observation excludes, and the observation counts only labels carrying
at least one node, because constraint DDL is schema machinery while
observation reports data. The guard tests compare the classpath's
schema objects against the live context registration, so an
unregistered schema fails the build here in place of drifting in
production.

Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
Registering a DiceStorageSchema bean now says on the interface what it
does: the schema's labels and relationship types become storage
bookkeeping the drift observation stops reporting, which is right for a
store's own nodes and wrong for domain data, whose home is the
DataDictionary. The guard fixture supplies a transaction manager so a
store bean that manages its own transactions can wire in the scanned
context, and a stale KDoc link follows the nodeShapes rename.

Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
The Drivine property bag is merge-only, so a release that dropped the two
quarantine keys from the proposition's metadata left previousStatus and reason
on the node. Saving a proposition now removes every metadata.* property the
saved metadata no longer names, with one dynamic REMOVE after the node write.
The drift integration test checks the keys are gone.

Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
The version now comes from the root pom's dependencyManagement.

Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com>
@jimador
jimador force-pushed the feat/metamodel-drift-store branch from 38e07f2 to 8004df5 Compare September 9, 2026 00:08

@igordayen igordayen left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jimador looks good!

@jimador
jimador merged commit 55c5bb4 into main Sep 9, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants