From 41d33c1539b7dc7c87ba33daebb184e0958b2fad Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Mon, 31 Aug 2026 09:13:26 -0400 Subject: [PATCH 1/7] Model durable extraction runs as pure values An ExtractionRun records what produced knowledge: tenant and run identity, profile, ordered source revisions, lineage with a denormalized root ref so whole-lineage reads never walk a chain, experiment and cohort refs, prompt, schema, and metamodel fingerprints, runtime identity, status, timing, counts, and bounded sanitized failures that can never carry source text. Requested and observed model facts are separate types with disjoint fields, so a guessed provider-effective value has nowhere to live. Invocation identity is the ordinal in a plan fixed before any call fires, and a failure can only reference an attempt the run has a record of. Every stored string is bounded and over-long values are rejected, not truncated; replay fidelity tops out at approximate, on purpose. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com> --- CHANGELOG.md | 106 +++++ .../extraction/ExtractionFailure.kt | 177 +++++++ .../extraction/ExtractionInvocationRecord.kt | 298 ++++++++++++ .../extraction/ExtractionOpaqueRef.kt | 152 ++++++ .../extraction/ExtractionReplayFidelity.kt | 70 +++ .../ExtractionRequestedModelConfig.kt | 118 +++++ .../proposition/extraction/ExtractionRun.kt | 327 +++++++++++++ .../extraction/ExtractionRunEnvelope.kt | 229 ++++++++++ .../extraction/ExtractionRunLimits.kt | 125 +++++ .../extraction/ExtractionRunLineage.kt | 120 +++++ .../extraction/ExtractionRunRef.kt | 68 +++ .../extraction/ExtractionRunStatus.kt | 73 +++ .../ExtractionInvocationIdentityTest.kt | 278 +++++++++++ .../extraction/ExtractionRunContractTest.kt | 432 ++++++++++++++++++ .../extraction/ExtractionRunFixtures.kt | 185 ++++++++ .../extraction/ExtractionRunLineageTest.kt | 155 +++++++ .../extraction/ExtractionRunPrivacyTest.kt | 279 +++++++++++ .../extraction/ExtractionRunValueTypesTest.kt | 233 ++++++++++ docs/design/INDEX.md | 8 + docs/design/extraction-runs.md | 320 +++++++++++++ 20 files changed, 3753 insertions(+) create mode 100644 dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionFailure.kt create mode 100644 dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionInvocationRecord.kt create mode 100644 dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionOpaqueRef.kt create mode 100644 dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionReplayFidelity.kt create mode 100644 dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRequestedModelConfig.kt create mode 100644 dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRun.kt create mode 100644 dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunEnvelope.kt create mode 100644 dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLimits.kt create mode 100644 dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLineage.kt create mode 100644 dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunRef.kt create mode 100644 dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunStatus.kt create mode 100644 dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionInvocationIdentityTest.kt create mode 100644 dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunContractTest.kt create mode 100644 dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunFixtures.kt create mode 100644 dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLineageTest.kt create mode 100644 dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunPrivacyTest.kt create mode 100644 dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunValueTypesTest.kt create mode 100644 docs/design/extraction-runs.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 345e7df3..9e763232 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1182,3 +1182,109 @@ and the consumer PRs that deliver it). slice's own code and tests used the parameter for anything, so there is nothing to migrate. When the reference does return it arrives as a field on `ExtractionRequest`, which is what the request object is for: the entry-point signatures will be the ones this entry describes. + slice's own code and tests used the parameter for anything, so there is nothing to migrate. + claimed for `SourceAnalysisContext`: two more fields rewrite `copy`, add two `componentN` + methods, and change the synthetic `$default` constructor, so Kotlin code compiled against an + earlier jar must be recompiled rather than swapped in — the same half of the boundary #64 + declined, pinned here by a test asserting exactly one `copy` remains and that it takes thirteen + arguments. No stored data changes and no migration is required: nothing serializes a profile or a + run reference yet. Extraction, resolution, and revision ordering are behaviour-identical; a + profile changes what a run is attributed to, not what it does. `ExtractionContentProfileRef` and + `ExtractionRunRef` both carry `@ApiStatus.Experimental` and their shapes may still move while + #67 lands. A Kotlin `@RequiresOptIn` marker would make that enforceable at the call site rather + than advisory; DICE defines none today and the design note records it as an open question. + +- **EXPERIMENTAL.** The extraction run model in `dice` core — the value types DICE #67's store, + lineage and wiring slices build on. `ExtractionRun`, keyed by (`ContextId`, `ExtractionRunRef`) + through the new `ExtractionRunKey`, records the profile version in force, the ordered source + revisions it read, its lineage, prompt/schema/metamodel fingerprints, extractor/host/runtime + identity, requested model configuration, pseudonymous subject references, experiment and cohort + labels, status, timing, counts, invocation records, bounded sanitized failures, and an explicit + replay-fidelity value. Nothing stores one yet: the lifecycle state machine and the store contract + are the next slice, and no code constructs a run during extraction until the wiring slice. + **Requested and observed model facts are separate types, structurally.** + `ExtractionRequestedModelConfig` sits on the run header and holds what the host asked for — + portable fields only: model and role, temperature, top-p, top-k, max tokens, presence and + frequency penalties, thinking and selection fingerprints, and a timeout. There is no provider + extension object, no settings blob and no free map, because that is where credentials, system + prompts and whole SDK request bodies get persisted by accident; a provider-specific knob folds + into one of the opaque fingerprints. What actually happened is an `ExtractionInvocationRecord` + per attempt, carrying the configured service, `ExtractionModelUsage`, + `ExtractionProviderResponseFacts`, timing and outcome. An invocation record has no field of the + requested type and the two share no property name — `requestedModel` versus `responseModel` — and + both are asserted by test, so nothing can present a setting as an observation. An absent observed + field stays absent: a run that asked for a model and got no model name back records null rather + than echoing the request. Ranges are validated where providers agree and left open where they do + not, so a temperature of 2.0 is accepted. **Invocation identity comes from the call plan.** + `ExtractionInvocationId` is (`invocationIndex`, `attempt`): the index is the call's ordinal in the + run's plan, allocated by `ExtractionInvocationRecord.plan(n)` before any request goes out, and the + attempt counts retries of that same call from 1. Completion order writes into identities that + already exist — there is no factory taking a position in a result list, `retry()` carries the + index forward and resets every observed field, `invocationsInPlanOrder()` reads the plan back out + of records stored in arrival order, and a run rejects two records sharing an identity. That makes + (`runId`, index, attempt) a deterministic child key for the store slices. **The root run + reference is denormalized.** `ExtractionRunLineage` carries the run, its parent, what it + supersedes, its pass index, and its root, following OpenLineage's `ParentRunFacet`, which also + ships a root alongside the immediate parent so consumers need not walk the chain. The root is + fixed at mint — a parentless run is its own root, a child takes its parent's root — and the + constructor rejects a parentless run whose root is another run, a child claiming to be its own + root, and self-parenting or self-supersession. Cycles longer than one need the other runs and stay + with the store that walks the chains. **Privacy is a contract with an enforced floor.** + `ExtractionActorRef`, `ExtractionRequestRef`, `ExtractionSessionRef`, + `ExtractionPersonalizationRef`, `ExtractionDeploymentRef`, `ExtractionExperimentRef` and + `ExtractionCohortRef` are all `ExtractionOpaqueRef`: host-minted tokens DICE compares and never + parses, bounded to 256 characters and restricted to `A-Z a-z 0-9 . _ : ~ -`, which rejects an + email address, a URL, a file path, a JSON fragment and a human name outright. The KDoc states what + that does not prove — a value type cannot tell a pseudonym from a username — rather than implying + a guarantee. A token's `toString` shows eight characters and a validation message never quotes the + value it rejected. `ExtractionFailure` is a classified `ExtractionFailureCode` plus a bounded + single-line detail; `fromThrowable`, the path DICE itself uses, never reads `Throwable.message` + and records exception class names down a bounded cycle-safe cause chain, because a provider quotes + the prompt back in its message. Tests extract from a fixture whose source text is known and assert + no fragment of it, no address shape, no link shape and no long digit run survives into a + field-by-field dump of a fully populated run. **Replay fidelity never claims exact replay.** + `ExtractionReplayFidelity` is `NONE`, `METADATA`, `APPROXIMATE`; the strongest value is still + approximate and `strongest()` returns it so appending a value cannot quietly strengthen the claim. + **`ExtractionRunStatus` ships the four values only** — `RUNNING`, `COMPLETED`, `FAILED`, + `CANCELLED` — with no transition rules, which belong to the store contract. `COMPLETED`'s meaning + is stated on the value: every product the run's request called for is durably persisted or + terminally disposed, written after persistence, so a run whose persistence never finished stays + `RUNNING`. The two MLflow states DICE does not have are deliberate: `SCHEDULED` has no writer + without a scheduler, and an externally killed run is `CANCELLED`, because stopping short of its + products is the same fact whichever side pressed stop. OpenTelemetry's GenAI attribute names are + not adopted: every `gen_ai.*` attribute is still at Development stability and the conventions + moved to a separate repository in June 2026, so pinning a stored schema to them buys interop now + and a migration later. **One cap rule**: every bound is a named constant on `ExtractionRunLimits`, + checked in the `init` block of the type that owns the value, and an over-long value is rejected + rather than truncated, because a shortened identifier is a different identifier. Identifiers cap + at 256 characters, source keys at 1024 (they come from `SourceLocator.key()` and can hold a long + URL), the one free-text failure detail at 512, and the three collections at 256 source revisions, + 1024 invocation records and 64 failures. The failure detail is the single exception to rejection + and only on the way in: its factories clip it, the constructor still rejects. `SourceRevisionRef` + predates the rule and validates non-blank only, so `ExtractionRun` applies the bound where it + stores one; moving the check onto that type is follow-up work. `ContextId.value` is the one + string a run stores that the rule does not cover — `ContextId` is a DICE-wide type owned by the + agent framework — which matters because the tenant is half of `ExtractionRunKey`, so the store + key is bounded on one side only. Two bounds are enforced away from the field they protect: + `plan(count)` checks the invocation limit against the count before allocating anything, so a + chunk-derived plan size cannot exhaust memory on its way to being rejected, and a run rejects a + failure whose `invocation` names an identity it holds no record of, because a dangling reference + reads as evidence about a call that nothing can join it to. A failure outside any model call + names no invocation and is always accepted. Timing on an invocation record is an observation and + may be absent on a terminal outcome: a `SUCCEEDED` attempt with no `startedAt` means the clock + was not recorded, and requiring one would push callers to invent a duration. Design note: + [docs/design/extraction-runs.md](docs/design/extraction-runs.md). + **Compatibility: additive, new types only.** Nothing existing changes. No existing class gains or + loses a member, no signature moves, no default changes, and no behaviour differs — this slice adds + types to `com.embabel.dice.proposition.extraction` and touches nothing that was already there. + Source, binary and Java compatibility are therefore all unaffected, and the scoped Kotlin ABI + boundary the Wave A and B slices declared does not apply because no existing data class gained a + field. No stored data changes and no migration is required: nothing serializes a run yet, and the + first thing that will is the store slice. `ExtractionRun` is a plain class rather than a data + class, so it publishes no `copy` or `componentN` to be compatible with later — deliberate, since a + data class cannot defensively copy its collection parameters and a seventeen-field generated + surface would pin an ABI while #67 is still moving; equality and hash are hand-written and a test + varies each of the seventeen components in turn. Every type carries `@ApiStatus.Experimental`, + asserted by a test that reads the class files because the annotation has class retention, and the + shapes may still move while the remaining #67 slices land. The `@RequiresOptIn` question #66 + raised is unchanged and still open. diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionFailure.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionFailure.kt new file mode 100644 index 00000000..09f51ce3 --- /dev/null +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionFailure.kt @@ -0,0 +1,177 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import org.jetbrains.annotations.ApiStatus +import java.time.Instant + +/** + * What kind of thing went wrong, from a fixed list. + * + * The code is what a query groups by and what an alert fires on, so it is an enum rather than a + * string a caller invents per site. Anything that does not fit is [INTERNAL] or [UNCLASSIFIED], + * and a code that keeps getting used for the wrong thing is a signal to add one. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + */ +@ApiStatus.Experimental +enum class ExtractionFailureCode { + + /** The model or its service could not be reached, or refused to serve the request. */ + MODEL_UNAVAILABLE, + + /** The call was still outstanding when the configured timeout expired. */ + MODEL_TIMEOUT, + + /** The provider declined to answer — a safety refusal, a content filter, a policy block. */ + MODEL_REFUSED, + + /** The provider rejected the call for quota or rate reasons. */ + RATE_LIMITED, + + /** A response arrived and could not be parsed into the expected shape. */ + DECODE_FAILED, + + /** A response parsed and then broke the schema or the metamodel it had to satisfy. */ + SCHEMA_VIOLATION, + + /** The source material could not be read at the revision the run was asked for. */ + SOURCE_UNAVAILABLE, + + /** Extraction produced results and storing them failed. */ + PERSISTENCE_FAILED, + + /** The run, or this part of it, was stopped before it finished. */ + CANCELLED, + + /** A defect on DICE's side of the boundary. */ + INTERNAL, + + /** Nothing above fits, and the caller would rather record the failure than force a code. */ + UNCLASSIFIED, +} + +/** + * One thing that went wrong during a run: a code, a short detail, when, and which invocation. + * + * A failure record is evidence, and evidence about a failure is where source text leaks. A model + * provider routinely quotes the prompt back in its exception message, a decode error carries the + * fragment it choked on, and both end up in a stored run header if someone writes + * `e.message` into one. So the DICE-minted path, [fromThrowable], never reads + * `Throwable.message` at all: it records the exception class names down the cause chain and + * nothing else, which cannot contain source text because it never touched any. + * + * [of] exists for the case where a caller genuinely knows something useful ("chunk 3 of 12 + * exceeded the token budget"). DICE cannot check what a caller puts there. It bounds it, flattens + * it to a single line so a pasted stack trace does not fit, and the contract is that the caller + * supplies a classification rather than a payload. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + * + * @property code How the failure is classified + * @property detail Short, single-line, sanitized explanation. Empty when the code says it all. + * @property at When the failure was recorded + * @property invocation The invocation and attempt this failure belongs to, or null for a failure + * that happened outside any model call + */ +@ApiStatus.Experimental +data class ExtractionFailure @JvmOverloads constructor( + val code: ExtractionFailureCode, + val detail: String = "", + val at: Instant = Instant.now(), + val invocation: ExtractionInvocationId? = null, +) { + + init { + require(detail.length <= ExtractionRunLimits.MAX_FAILURE_DETAIL_LENGTH) { + "detail must be at most ${ExtractionRunLimits.MAX_FAILURE_DETAIL_LENGTH} characters, " + + "was ${detail.length}" + } + require(detail.none { it == '\n' || it == '\r' }) { + "detail must be a single line; a multi-line detail is a stack trace or a quoted payload" + } + } + + companion object { + + /** How many links of a cause chain [fromThrowable] walks. */ + const val MAX_CAUSE_CHAIN: Int = 5 + + /** + * Records a failure with a detail the caller wrote. + * + * Whitespace collapses to single spaces and the result is clipped to + * [ExtractionRunLimits.MAX_FAILURE_DETAIL_LENGTH]. The caller is responsible for the + * detail holding no source text, no prompt, and no personal data; nothing here can verify + * that. [fromThrowable] is the path to use when the detail would have come from an + * exception. + */ + @JvmStatic + @JvmOverloads + fun of( + code: ExtractionFailureCode, + detail: String, + at: Instant = Instant.now(), + invocation: ExtractionInvocationId? = null, + ): ExtractionFailure = ExtractionFailure( + code = code, + detail = sanitize(detail), + at = at, + invocation = invocation, + ) + + /** + * Records a failure from a throwable, using its class names and nothing else. + * + * The detail is the exception class name, then up to [MAX_CAUSE_CHAIN] causes joined by + * ` <- `. `Throwable.message`, suppressed exceptions, and the stack trace are all + * untouched, so the run header cannot pick up the prompt, the response body, or the + * fragment that failed to parse. + * + * A host that wants the message keeps it in its own logs, where retention and access are + * its to set. + */ + @JvmStatic + @JvmOverloads + fun fromThrowable( + code: ExtractionFailureCode, + throwable: Throwable, + at: Instant = Instant.now(), + invocation: ExtractionInvocationId? = null, + ): ExtractionFailure = ExtractionFailure( + code = code, + detail = sanitize(causeChain(throwable).joinToString(" <- ")), + at = at, + invocation = invocation, + ) + + private fun causeChain(throwable: Throwable): List { + val names = mutableListOf() + var current: Throwable? = throwable + val seen = mutableSetOf() + while (current != null && names.size < MAX_CAUSE_CHAIN && seen.add(current)) { + names += current.javaClass.name + current = current.cause + } + return names + } + + private fun sanitize(detail: String): String = + detail.replace(WHITESPACE, " ").trim().take(ExtractionRunLimits.MAX_FAILURE_DETAIL_LENGTH) + + private val WHITESPACE = Regex("\\s+") + } +} diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionInvocationRecord.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionInvocationRecord.kt new file mode 100644 index 00000000..d4e03106 --- /dev/null +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionInvocationRecord.kt @@ -0,0 +1,298 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import org.jetbrains.annotations.ApiStatus +import java.time.Instant + +/** + * The stable identity of one model call within a run: which call, and which try at it. + * + * [invocationIndex] is the call's ordinal in the run's call plan, and it is allocated when the + * plan is laid out — before any call goes out. Two runs that chunk the same material the same way + * give the same piece of work the same index, whichever finishes first. [attempt] counts tries at + * that same call, starting at 1. + * + * Nothing derives an identity from completion order. There is no factory that takes a position in + * a result list, and parallel execution writes into identities that already exist rather than + * minting them as answers arrive. That is what makes (`runId`, index, attempt) a deterministic + * child key: a retried write lands on its own row and a replayed write upserts in place. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + * + * @property invocationIndex Ordinal of this call in the run's plan, counting from zero + * @property attempt Which try at that call this is, counting from one + */ +@ApiStatus.Experimental +data class ExtractionInvocationId( + val invocationIndex: Int, + val attempt: Int, +) { + + init { + require(invocationIndex >= 0) { "invocationIndex must not be negative, was $invocationIndex" } + require(attempt >= 1) { "attempt must be at least 1, was $attempt" } + } + + /** The identity of the next try at the same call. */ + fun nextAttempt(): ExtractionInvocationId = copy(attempt = attempt + 1) + + override fun toString(): String = "invocation $invocationIndex attempt $attempt" + + companion object { + + /** + * The identity of the first try at the call at [invocationIndex] in the plan. + */ + @JvmStatic + fun planned(invocationIndex: Int): ExtractionInvocationId = + ExtractionInvocationId(invocationIndex = invocationIndex, attempt = 1) + } +} + +/** + * How one attempt at one model call ended. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + */ +@ApiStatus.Experimental +enum class ExtractionInvocationOutcome { + + /** The identity exists and the call has not come back. This is what a planned call starts as. */ + IN_FLIGHT, + + /** A response arrived and was usable. */ + SUCCEEDED, + + /** The attempt failed. The run's failure records say how. */ + FAILED, + + /** The attempt was stopped before it produced anything. */ + CANCELLED, +} + +/** + * Tokens a call actually consumed, as the provider reported them. + * + * Recorded as reported. DICE does not recompute [totalTokens] from the other two or reconcile + * them when a provider's arithmetic looks off, because the point of an observed record is what + * was observed. Providers count cached and reasoning tokens differently and some report neither; + * absent means the provider said nothing, not zero. + * + * Native usage objects are not stored. These are the portable counts, pulled out of whatever + * shape the SDK returned. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + * + * @property inputTokens Tokens the provider counted on the way in + * @property outputTokens Tokens the provider counted on the way out + * @property totalTokens The provider's own total, when it gave one + * @property cachedInputTokens Input tokens the provider served from its cache + * @property reasoningTokens Output tokens the provider attributed to reasoning + */ +@ApiStatus.Experimental +data class ExtractionModelUsage @JvmOverloads constructor( + val inputTokens: Int? = null, + val outputTokens: Int? = null, + val totalTokens: Int? = null, + val cachedInputTokens: Int? = null, + val reasoningTokens: Int? = null, +) { + + init { + requireNonNegative(inputTokens, "inputTokens") + requireNonNegative(outputTokens, "outputTokens") + requireNonNegative(totalTokens, "totalTokens") + requireNonNegative(cachedInputTokens, "cachedInputTokens") + requireNonNegative(reasoningTokens, "reasoningTokens") + } + + companion object { + + /** Java-friendly factory. */ + @JvmStatic + @JvmOverloads + fun of( + inputTokens: Int? = null, + outputTokens: Int? = null, + totalTokens: Int? = null, + cachedInputTokens: Int? = null, + reasoningTokens: Int? = null, + ): ExtractionModelUsage = ExtractionModelUsage( + inputTokens = inputTokens, + outputTokens = outputTokens, + totalTokens = totalTokens, + cachedInputTokens = cachedInputTokens, + reasoningTokens = reasoningTokens, + ) + } +} + +/** + * What the provider said about its own response. + * + * Every field is present only when the provider reported it. None of it is inferred from + * [ExtractionRequestedModelConfig]: a run that asked for `gpt-x` and got no model name back + * records a null [responseModel] rather than echoing what it asked for, because the whole point + * of the field is telling those two cases apart. + * + * Response bodies, message content, and SDK objects are not stored — these are the identifiers + * and the classification, which is what an incident needs to correlate with the provider's own + * logs. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + * + * @property responseModel The model the provider says answered + * @property responseId The provider's id for the response, for correlating with its support logs + * @property finishReason Why the provider stopped generating, in its own vocabulary + * @property systemFingerprint The provider's fingerprint for the backend configuration that served + * the call + */ +@ApiStatus.Experimental +data class ExtractionProviderResponseFacts @JvmOverloads constructor( + val responseModel: String? = null, + val responseId: String? = null, + val finishReason: String? = null, + val systemFingerprint: String? = null, +) { + + init { + requireBoundedIdentifier(responseModel, "responseModel") + requireBoundedIdentifier(responseId, "responseId") + requireBoundedIdentifier(finishReason, "finishReason") + requireBoundedIdentifier(systemFingerprint, "systemFingerprint") + } + + companion object { + + /** Java-friendly factory. */ + @JvmStatic + @JvmOverloads + fun of( + responseModel: String? = null, + responseId: String? = null, + finishReason: String? = null, + systemFingerprint: String? = null, + ): ExtractionProviderResponseFacts = ExtractionProviderResponseFacts( + responseModel = responseModel, + responseId = responseId, + finishReason = finishReason, + systemFingerprint = systemFingerprint, + ) + } +} + +/** + * One attempt at one model call: its identity, and what was observed about it. + * + * Observed facts only. There is no requested-configuration field on this type and there will not + * be one — that separation is the whole reason [ExtractionRequestedModelConfig] is a different + * type sitting on the run rather than a section of this one. A reader of an invocation record can + * never mistake "what we asked for" for "what happened", because the record cannot express the + * first. + * + * [configuredService] is an observed fact too: it is the service this attempt was actually + * dispatched against, which a router can change between attempts. + * + * Failures live on the run's bounded failure list, tagged with this record's [id], rather than + * being duplicated here — one bound, one place to sanitize. A run rejects a failure naming an + * attempt it has no record of, so the pair arrives together. + * + * **Timing is an observation and may be absent on a terminal record.** A `SUCCEEDED` attempt with + * no [startedAt] is constructible, and means the clock was not recorded rather than that the call + * did not run. Requiring timing would push callers to invent it, and a made-up duration is worse + * evidence than none. The two checks that do apply are the ones a record can be wrong about on its + * own terms: a finish cannot precede its start, and an [ExtractionInvocationOutcome.IN_FLIGHT] + * attempt has not finished. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + * + * @property id Which call and which attempt this record is + * @property outcome How the attempt ended + * @property configuredService The service this attempt went to, as configured at dispatch + * @property startedAt When the attempt was dispatched, or null if it never was + * @property finishedAt When the attempt came back + * @property usage Tokens the provider reported + * @property providerResponse What the provider said about its response + */ +@ApiStatus.Experimental +data class ExtractionInvocationRecord @JvmOverloads constructor( + val id: ExtractionInvocationId, + val outcome: ExtractionInvocationOutcome = ExtractionInvocationOutcome.IN_FLIGHT, + val configuredService: String? = null, + val startedAt: Instant? = null, + val finishedAt: Instant? = null, + val usage: ExtractionModelUsage? = null, + val providerResponse: ExtractionProviderResponseFacts? = null, +) { + + init { + requireBoundedIdentifier(configuredService, "configuredService") + if (finishedAt != null) { + require(startedAt != null) { "an attempt cannot finish without having started" } + require(!finishedAt.isBefore(startedAt)) { "finishedAt must not be before startedAt" } + } + require(outcome != ExtractionInvocationOutcome.IN_FLIGHT || finishedAt == null) { + "an IN_FLIGHT attempt has no finishedAt" + } + } + + /** Ordinal of this call in the run's plan. */ + val invocationIndex: Int + get() = id.invocationIndex + + /** Which try at that call this record is. */ + val attempt: Int + get() = id.attempt + + /** + * The record for the next try at this same call. + * + * The index carries over, the attempt increments, and every observed field resets, because + * the observations belonged to the attempt that just failed. + */ + fun retry(): ExtractionInvocationRecord = ExtractionInvocationRecord(id = id.nextAttempt()) + + companion object { + + /** + * The record of a call that the plan has allocated and nothing has dispatched yet. + * + * Call this while laying out the plan, before the first request goes out, so the identity + * exists before any answer can suggest one. + */ + @JvmStatic + fun planned(invocationIndex: Int): ExtractionInvocationRecord = + ExtractionInvocationRecord(id = ExtractionInvocationId.planned(invocationIndex)) + + /** + * Lays out a whole call plan: [count] invocations, indices 0 to `count - 1`, all on their + * first attempt and none dispatched. + * + * The count is checked against [ExtractionRunLimits.MAX_INVOCATIONS] before anything is + * allocated. A plan size derived from chunking a large document can be enormous, and + * finding that out from the run's own bound would mean building the whole list first. + */ + @JvmStatic + fun plan(count: Int): List { + require(count >= 0) { "count must not be negative, was $count" } + require(count <= ExtractionRunLimits.MAX_INVOCATIONS) { + "a call plan may hold at most ${ExtractionRunLimits.MAX_INVOCATIONS} invocations, was $count" + } + return (0 until count).map { planned(it) } + } + } +} diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionOpaqueRef.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionOpaqueRef.kt new file mode 100644 index 00000000..216a80a2 --- /dev/null +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionOpaqueRef.kt @@ -0,0 +1,152 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import org.jetbrains.annotations.ApiStatus + +/** + * A bounded, host-minted token that names something about a run without describing it. + * + * DICE compares these and stores them and parses nothing out of them. Every one of them exists so + * an audit can ask "which runs share this actor?" or "which runs ran under this deployment?" + * without DICE holding a user object, a session object, or a personalization payload. + * + * **The contract a host takes on when it mints one:** + * - It is a pseudonym. Not an email address, not a username, not a phone number, not a customer + * number, not a name — nothing that identifies a person on its own. + * - It is not dereferenceable into anything sensitive. Not a URL, not a signed link, not a bearer + * token, not an API key, not a session cookie value. + * - It carries no authorization. Holding one grants nothing; DICE never presents it to anything. + * - It is stable enough to group by and cheap enough to rotate. A host that wants to break the + * link between a subject and its past runs rotates the token, and the old runs stay grouped + * among themselves. + * + * **What the type can enforce, and what it cannot.** Construction bounds the length and restricts + * the characters to `A-Z a-z 0-9 . _ : ~ -`, which rules out whitespace, control characters, + * `@`, `/` and `\` — so an email address, a URL, a file path and a human name are all rejected + * outright, and the common shapes of a leaked identifier cannot be stored. It cannot tell a + * pseudonym from a username, or a random token from a customer number: `jdunnam` and `55512345` + * both pass. The last mile of that contract is the host's, and the design note says so in the same + * words. + * + * [toString] shows only the first [TOKEN_PREVIEW_LENGTH] characters, so a token does not spread + * through logs and exception messages in full. Validation messages never quote the token at all. + * + * Equality is by exact type and token together, so an actor token and a session token that happen + * to hold the same string are two different references. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + * + * @property token The opaque value the host minted + */ +@ApiStatus.Experimental +sealed class ExtractionOpaqueRef(val token: String) { + + init { + require(token.isNotBlank()) { "${javaClass.simpleName} token must not be blank" } + require(token.length <= ExtractionRunLimits.MAX_IDENTIFIER_LENGTH) { + "${javaClass.simpleName} token must be at most " + + "${ExtractionRunLimits.MAX_IDENTIFIER_LENGTH} characters, was ${token.length}" + } + require(TOKEN_PATTERN.matches(token)) { + "${javaClass.simpleName} token must contain only letters, digits and . _ : ~ - " + + "so that an address, a URL, a path or a name cannot be stored as one" + } + } + + final override fun equals(other: Any?): Boolean { + if (this === other) return true + if (other == null || javaClass != other.javaClass) return false + return token == (other as ExtractionOpaqueRef).token + } + + final override fun hashCode(): Int = 31 * javaClass.hashCode() + token.hashCode() + + final override fun toString(): String = + "${javaClass.simpleName}(token=${token.take(TOKEN_PREVIEW_LENGTH)}…)" + + companion object { + + /** Characters an opaque token may contain. */ + private val TOKEN_PATTERN = Regex("[A-Za-z0-9._:~-]+") + + /** How much of a token [toString] shows: enough to correlate two lines, not the value. */ + const val TOKEN_PREVIEW_LENGTH: Int = 8 + } +} + +/** + * Names whoever the run acted for, pseudonymously. + * + * This is the reference an audit groups by to answer "what was extracted on this person's + * behalf?" without DICE ever holding who that person is. + */ +@ApiStatus.Experimental +class ExtractionActorRef(token: String) : ExtractionOpaqueRef(token) + +/** + * Names the inbound request the run was started for. + * + * A host that already carries a request or correlation id through its own logs passes the same one + * here, so a run lines up with the request that caused it. + */ +@ApiStatus.Experimental +class ExtractionRequestRef(token: String) : ExtractionOpaqueRef(token) + +/** + * Names the conversation or session the material came from. + * + * Several runs over one long conversation share this reference, which is what makes "everything + * extracted from that session" a single query. + */ +@ApiStatus.Experimental +class ExtractionSessionRef(token: String) : ExtractionOpaqueRef(token) + +/** + * Names the personalization state in force, without holding any of it. + * + * A host that varies extraction by user preferences, memory, or tuned instructions fingerprints + * that state and passes the fingerprint. Two runs with the same reference ran under the same + * personalization; what it contained stays with the host. + */ +@ApiStatus.Experimental +class ExtractionPersonalizationRef(token: String) : ExtractionOpaqueRef(token) + +/** + * Names the deployment the run executed in — an environment, a region, a release, a shard. + * + * This is what separates "the model got worse" from "the model got worse in one deployment". + */ +@ApiStatus.Experimental +class ExtractionDeploymentRef(token: String) : ExtractionOpaqueRef(token) + +/** + * Names the experiment a run belongs to. + * + * A label to group and compare by. It carries the same no-personal-data contract as the rest of + * the family, because an experiment name is a place where a description of the subjects tends to + * end up. + */ +@ApiStatus.Experimental +class ExtractionExperimentRef(token: String) : ExtractionOpaqueRef(token) + +/** + * Names the arm or cohort within an experiment. + * + * Same contract as [ExtractionExperimentRef]: a label, not a description of who is in it. + */ +@ApiStatus.Experimental +class ExtractionCohortRef(token: String) : ExtractionOpaqueRef(token) diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionReplayFidelity.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionReplayFidelity.kt new file mode 100644 index 00000000..f969edcf --- /dev/null +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionReplayFidelity.kt @@ -0,0 +1,70 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import org.jetbrains.annotations.ApiStatus + +/** + * How much of a run someone could set up again from what the run recorded. + * + * There is no value here that means "run this again and get the same output", and there will not + * be one. A hosted model can change weights, quantization, routing, safety filtering, and system + * instructions under a stable model name, and none of that is visible to DICE. Temperature zero + * narrows the distribution and does not remove batching and floating-point nondeterminism. So the + * strongest value the model offers is [APPROXIMATE], and it is named for what it is. + * + * The field says what the *record* supports. It is not a promise about the provider, and a host + * replay policy — how many attempts, against which provider, with what tolerance — stays the + * host's. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + */ +@ApiStatus.Experimental +enum class ExtractionReplayFidelity { + + /** + * The run recorded nothing that would help set it up again. It can be counted and attributed; + * it cannot be re-run in any recognisable form. + */ + NONE, + + /** + * Identities and fingerprints only. You can tell which prompt template, schema, metamodel and + * profile version were in play, and compare two runs by those, without being able to + * reconstruct the input to either. + */ + METADATA, + + /** + * Identities, fingerprints, and the requested model configuration. A host can stand up a + * similar run against the same declared model and settings. The output will differ, sometimes + * materially, and any comparison between the two is a comparison of two runs rather than a + * verification of one. + */ + APPROXIMATE, + ; + + companion object { + + /** + * The strongest fidelity DICE records, which is still approximate. Call this rather than + * taking the last enum entry, so the honesty of the claim survives someone appending a + * value. + */ + @JvmStatic + fun strongest(): ExtractionReplayFidelity = APPROXIMATE + } +} diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRequestedModelConfig.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRequestedModelConfig.kt new file mode 100644 index 00000000..2a00767a --- /dev/null +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRequestedModelConfig.kt @@ -0,0 +1,118 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import org.jetbrains.annotations.ApiStatus +import java.time.Duration + +/** + * What the run *asked* for. + * + * Everything here was decided before any call went out. Nothing here is evidence that a provider + * honoured it: a service can clamp a temperature, ignore a top-k, silently route to a different + * checkpoint, or cap max tokens below what was asked. What actually came back is + * [ExtractionProviderResponseFacts], on the invocation record, and the two are separate types so + * that no field can hold both meanings and no mapper can drift one into the other. They share no + * property name, which is why the requested model is [requestedModel] and the reported one is + * `responseModel`. + * + * **Portable fields only.** Every field here means the same thing across providers. There is no + * extension object, no provider settings blob, and no free map, because that is where credentials, + * system prompts, and whole SDK request bodies get persisted by accident. A provider-specific + * knob that matters to a host is folded into [selectionFingerprint] or [thinkingFingerprint] — + * an opaque digest DICE compares and never reads. + * + * Ranges are checked where every provider agrees and left open where they do not. Temperature has + * no upper bound here because services differ on whether it stops at 1 or 2; the penalties are + * only required to be real numbers for the same reason. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + * + * @property modelRole The host's name for the job this model was doing, such as `extraction` + * @property requestedModel The model name the host asked for + * @property temperature Requested sampling temperature + * @property topP Requested nucleus-sampling mass, between 0 and 1 + * @property topK Requested top-k cutoff + * @property maxTokens Requested output-token ceiling + * @property presencePenalty Requested presence penalty + * @property frequencyPenalty Requested frequency penalty + * @property thinkingFingerprint Opaque digest of the reasoning or thinking configuration + * @property selectionFingerprint Opaque digest of how the host chose this model and these settings + * @property timeout How long the caller was prepared to wait + */ +@ApiStatus.Experimental +data class ExtractionRequestedModelConfig @JvmOverloads constructor( + val modelRole: String? = null, + val requestedModel: String? = null, + val temperature: Double? = null, + val topP: Double? = null, + val topK: Int? = null, + val maxTokens: Int? = null, + val presencePenalty: Double? = null, + val frequencyPenalty: Double? = null, + val thinkingFingerprint: String? = null, + val selectionFingerprint: String? = null, + val timeout: Duration? = null, +) { + + init { + requireBoundedIdentifier(modelRole, "modelRole") + requireBoundedIdentifier(requestedModel, "requestedModel") + requireBoundedIdentifier(thinkingFingerprint, "thinkingFingerprint") + requireBoundedIdentifier(selectionFingerprint, "selectionFingerprint") + requireAtLeast(temperature, "temperature", 0.0) + requireInRange(topP, "topP", 0.0, 1.0) + requireFinite(presencePenalty, "presencePenalty") + requireFinite(frequencyPenalty, "frequencyPenalty") + if (topK != null) require(topK >= 1) { "topK must be at least 1, was $topK" } + if (maxTokens != null) require(maxTokens >= 1) { "maxTokens must be at least 1, was $maxTokens" } + if (timeout != null) { + require(!timeout.isZero && !timeout.isNegative) { "timeout must be positive, was $timeout" } + } + } + + companion object { + + /** Java-friendly factory. */ + @JvmStatic + @JvmOverloads + fun of( + modelRole: String? = null, + requestedModel: String? = null, + temperature: Double? = null, + topP: Double? = null, + topK: Int? = null, + maxTokens: Int? = null, + presencePenalty: Double? = null, + frequencyPenalty: Double? = null, + thinkingFingerprint: String? = null, + selectionFingerprint: String? = null, + timeout: Duration? = null, + ): ExtractionRequestedModelConfig = ExtractionRequestedModelConfig( + modelRole = modelRole, + requestedModel = requestedModel, + temperature = temperature, + topP = topP, + topK = topK, + maxTokens = maxTokens, + presencePenalty = presencePenalty, + frequencyPenalty = frequencyPenalty, + thinkingFingerprint = thinkingFingerprint, + selectionFingerprint = selectionFingerprint, + timeout = timeout, + ) + } +} diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRun.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRun.kt new file mode 100644 index 00000000..be17b595 --- /dev/null +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRun.kt @@ -0,0 +1,327 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import com.embabel.agent.core.ContextId +import com.embabel.dice.provenance.SourceRevisionRef +import org.jetbrains.annotations.ApiStatus +import java.time.Instant +import java.util.Collections + +/** + * Identifies one run inside one tenant. + * + * A run id is host-minted and DICE never assumes it is globally unique, so the tenant travels with + * it everywhere. Two tenants that both mint the run id `run-1` have two different runs, and this + * type is why nothing can accidentally treat them as one. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + * + * @property contextId The tenant that owns the run + * @property runRef The run + */ +@ApiStatus.Experimental +data class ExtractionRunKey( + val contextId: ContextId, + val runRef: ExtractionRunRef, +) { + + /** The tenant id as a plain string, for Java callers, since `ContextId` is a value class. */ + fun getContextIdValue(): String = contextId.value +} + +/** + * The durable record of one extraction execution. + * + * A run answers "what produced this claim, under what, and how did it go?" — the profile and + * prompt and schema versions in force, which revisions of which sources were read, which model + * was asked for and what the provider reported back, how far it got, and what went wrong. It is + * the header; the propositions it produced are attributed to it by a separate relation, and source + * grounding stays what it was. + * + * **Two rules shape the type.** + * + * The first is that requested and observed are different types. What a run asked a model for is + * [requestedModel], one [ExtractionRequestedModelConfig] on the header. What actually happened is + * an [ExtractionInvocationRecord] per attempt, holding usage, timing, the service it went to, and + * whatever the provider reported. An invocation record has no field that can hold a requested + * value, so nothing can quietly present a setting as an observation. + * + * The second is that the run holds no content. No prompts, no source text, no responses, no + * user or session objects, no provider SDK payloads, no extension maps. What a host would have + * needed those for is covered by digests it can compare ([fingerprints]) and by bounded + * pseudonymous tokens it can group by ([subjectRefs]). Failures are classified codes with a short + * sanitized detail, and the path DICE itself uses never reads an exception message. + * + * **What is not decided here.** Which status transitions are legal, which are compare-and-set, and + * what a store does with a repeated terminal write belong to the run store contract in the next + * slice. This type checks that a finish does not precede a start and stops there; it does not + * require, for instance, that a [ExtractionRunStatus.COMPLETED] run has a [finishedAt], so the + * state machine defines that once instead of twice. + * + * Collections are copied on the way in, always, including empty ones — an empty mutable list a + * caller keeps a handle on is the same aliasing bug as a full one, and it fails later and stranger. + * The copies are unmodifiable, so the run a caller reads back cannot be edited through the list + * either. + * + * This is a plain class rather than a data class on purpose. A data class has to declare its + * collection parameters as properties, which means the field is the caller's list and there is + * nowhere to copy it; its generated `copy` and `componentN` methods would also pin an ABI across + * seventeen fields while #67 is still moving. Equality and hash are written out over every + * component instead. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + * + * @property contextId The tenant that owns this run + * @property lineage This run's reference, its root, its parent, what it supersedes, and its pass + * @property status Where the run stands + * @property startedAt When the run began + * @property finishedAt When it reached a terminal state, or null while it has not + * @property profile The content profile version in force + * @property sourceRevisions Which revisions of which sources were read, in the order they were read + * @property fingerprints Digests of the prompt, schema and metamodel in force + * @property runtime What code ran it, and where + * @property requestedModel What the run asked a model for + * @property subjectRefs Pseudonymous references to whose work this was + * @property experimentRef The experiment this run belongs to + * @property cohortRef The arm within that experiment + * @property replayFidelity How much of this run someone could set up again from what it recorded + * @property counts How much the run got through + * @property invocations One record per attempt at each planned model call + * @property failures Bounded, sanitized record of what went wrong + */ +@ApiStatus.Experimental +class ExtractionRun @JvmOverloads constructor( + val contextId: ContextId, + val lineage: ExtractionRunLineage, + val status: ExtractionRunStatus, + val startedAt: Instant, + val finishedAt: Instant? = null, + val profile: ExtractionContentProfileRef? = null, + sourceRevisions: List = emptyList(), + val fingerprints: ExtractionRunFingerprints = ExtractionRunFingerprints(), + val runtime: ExtractionRuntimeIdentity = ExtractionRuntimeIdentity(), + val requestedModel: ExtractionRequestedModelConfig? = null, + val subjectRefs: ExtractionRunSubjectRefs = ExtractionRunSubjectRefs(), + val experimentRef: ExtractionExperimentRef? = null, + val cohortRef: ExtractionCohortRef? = null, + val replayFidelity: ExtractionReplayFidelity = ExtractionReplayFidelity.NONE, + val counts: ExtractionRunCounts = ExtractionRunCounts(), + invocations: List = emptyList(), + failures: List = emptyList(), +) { + + /** Which revisions of which sources this run read, in order. */ + val sourceRevisions: List = + Collections.unmodifiableList(ArrayList(sourceRevisions)) + + /** One record per attempt at each planned call, in whatever order the caller supplied. */ + val invocations: List = + Collections.unmodifiableList(ArrayList(invocations)) + + /** What went wrong, bounded and sanitized. */ + val failures: List = + Collections.unmodifiableList(ArrayList(failures)) + + init { + require(finishedAt == null || !finishedAt.isBefore(startedAt)) { + "finishedAt must not be before startedAt" + } + + require(this.sourceRevisions.size <= ExtractionRunLimits.MAX_SOURCE_REVISIONS) { + "a run may record at most ${ExtractionRunLimits.MAX_SOURCE_REVISIONS} source revisions, " + + "was ${this.sourceRevisions.size}" + } + require(this.sourceRevisions.distinct().size == this.sourceRevisions.size) { + "sourceRevisions must be distinct; a run reads each source revision once" + } + // SourceRevisionRef predates the run model's cap rule and validates non-blank only, so the + // bound is applied where the run stores it. Moving it onto the type is a follow-up. + this.sourceRevisions.forEach { revision -> + require(revision.sourceKey.length <= ExtractionRunLimits.MAX_SOURCE_KEY_LENGTH) { + "sourceKey must be at most ${ExtractionRunLimits.MAX_SOURCE_KEY_LENGTH} characters, " + + "was ${revision.sourceKey.length}" + } + require(revision.sourceRevision.length <= ExtractionRunLimits.MAX_IDENTIFIER_LENGTH) { + "sourceRevision must be at most ${ExtractionRunLimits.MAX_IDENTIFIER_LENGTH} characters, " + + "was ${revision.sourceRevision.length}" + } + } + + require(this.invocations.size <= ExtractionRunLimits.MAX_INVOCATIONS) { + "a run may record at most ${ExtractionRunLimits.MAX_INVOCATIONS} invocation records, " + + "was ${this.invocations.size}" + } + val identities = this.invocations.map { it.id } + require(identities.distinct().size == identities.size) { + "invocation records must have distinct (invocationIndex, attempt) identities" + } + + require(this.failures.size <= ExtractionRunLimits.MAX_FAILURES) { + "a run may record at most ${ExtractionRunLimits.MAX_FAILURES} failures, was ${this.failures.size}" + } + // A failure that names an attempt the run has no record of is a dangling audit reference: + // it reads as evidence about a call, and nothing can join it to one. Rejected here so the + // pair arrives together or not at all. + val identitySet = identities.toSet() + this.failures.forEach { failure -> + val invocation = failure.invocation ?: return@forEach + require(invocation in identitySet) { + "a failure names $invocation, which this run has no invocation record for" + } + } + } + + /** This run's reference. */ + val ref: ExtractionRunRef + get() = lineage.runRef + + /** The oldest run in this run's lineage, which is [ref] itself when this run has no parent. */ + val rootRef: ExtractionRunRef + get() = lineage.rootRunRef + + /** The run this one continues from, or null. */ + val parentRef: ExtractionRunRef? + get() = lineage.parentRunRef + + /** True when this run starts its lineage. */ + val isRoot: Boolean + get() = lineage.isRoot + + /** The tenant-qualified identity a store keys this run on. */ + fun key(): ExtractionRunKey = ExtractionRunKey(contextId, ref) + + /** The tenant id as a plain string, for Java callers, since `ContextId` is a value class. */ + fun getContextIdValue(): String = contextId.value + + /** + * The invocation records ordered by the plan: call 0 before call 1, and within a call, first + * attempt before second. + * + * The order records were handed to the constructor is the order calls came back, which is not + * the order they were planned in. This reads the identities that were allocated up front. + */ + fun invocationsInPlanOrder(): List = + invocations.sortedWith(compareBy({ it.invocationIndex }, { it.attempt })) + + /** Every attempt at the call at [invocationIndex], earliest attempt first. */ + fun attemptsOf(invocationIndex: Int): List = + invocations.filter { it.invocationIndex == invocationIndex }.sortedBy { it.attempt } + + override fun equals(other: Any?): Boolean { + if (this === other) return true + if (other !is ExtractionRun) return false + return contextId == other.contextId && + lineage == other.lineage && + status == other.status && + startedAt == other.startedAt && + finishedAt == other.finishedAt && + profile == other.profile && + sourceRevisions == other.sourceRevisions && + fingerprints == other.fingerprints && + runtime == other.runtime && + requestedModel == other.requestedModel && + subjectRefs == other.subjectRefs && + experimentRef == other.experimentRef && + cohortRef == other.cohortRef && + replayFidelity == other.replayFidelity && + counts == other.counts && + invocations == other.invocations && + failures == other.failures + } + + override fun hashCode(): Int { + var result = contextId.hashCode() + result = 31 * result + lineage.hashCode() + result = 31 * result + status.hashCode() + result = 31 * result + startedAt.hashCode() + result = 31 * result + (finishedAt?.hashCode() ?: 0) + result = 31 * result + (profile?.hashCode() ?: 0) + result = 31 * result + sourceRevisions.hashCode() + result = 31 * result + fingerprints.hashCode() + result = 31 * result + runtime.hashCode() + result = 31 * result + (requestedModel?.hashCode() ?: 0) + result = 31 * result + subjectRefs.hashCode() + result = 31 * result + (experimentRef?.hashCode() ?: 0) + result = 31 * result + (cohortRef?.hashCode() ?: 0) + result = 31 * result + replayFidelity.hashCode() + result = 31 * result + counts.hashCode() + result = 31 * result + invocations.hashCode() + result = 31 * result + failures.hashCode() + return result + } + + /** + * A summary: identity, lineage, state, and sizes. + * + * It leaves out the digests, the reference tokens and the failure details, so a run logged at + * an error site does not spread them. Anything that needs every field reads the properties. + */ + override fun toString(): String = + "ExtractionRun(contextId=${contextId.value}, runId=${ref.runId}, rootRunId=${rootRef.runId}, " + + "parentRunId=${parentRef?.runId}, pass=${lineage.passIndex}, status=$status, " + + "startedAt=$startedAt, finishedAt=$finishedAt, sourceRevisions=${sourceRevisions.size}, " + + "invocations=${invocations.size}, failures=${failures.size}, replayFidelity=$replayFidelity)" + + companion object { + + /** + * Java-friendly factory taking the tenant as a plain string. + * + * `ContextId` is a Kotlin value class, so a factory that took one directly would have a + * mangled JVM name. Kotlin callers can use the constructor. + */ + @JvmStatic + @JvmOverloads + fun of( + contextIdValue: String, + lineage: ExtractionRunLineage, + status: ExtractionRunStatus, + startedAt: Instant, + finishedAt: Instant? = null, + profile: ExtractionContentProfileRef? = null, + sourceRevisions: List = emptyList(), + fingerprints: ExtractionRunFingerprints = ExtractionRunFingerprints(), + runtime: ExtractionRuntimeIdentity = ExtractionRuntimeIdentity(), + requestedModel: ExtractionRequestedModelConfig? = null, + subjectRefs: ExtractionRunSubjectRefs = ExtractionRunSubjectRefs(), + experimentRef: ExtractionExperimentRef? = null, + cohortRef: ExtractionCohortRef? = null, + replayFidelity: ExtractionReplayFidelity = ExtractionReplayFidelity.NONE, + counts: ExtractionRunCounts = ExtractionRunCounts(), + invocations: List = emptyList(), + failures: List = emptyList(), + ): ExtractionRun = ExtractionRun( + contextId = ContextId(contextIdValue), + lineage = lineage, + status = status, + startedAt = startedAt, + finishedAt = finishedAt, + profile = profile, + sourceRevisions = sourceRevisions, + fingerprints = fingerprints, + runtime = runtime, + requestedModel = requestedModel, + subjectRefs = subjectRefs, + experimentRef = experimentRef, + cohortRef = cohortRef, + replayFidelity = replayFidelity, + counts = counts, + invocations = invocations, + failures = failures, + ) + } +} diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunEnvelope.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunEnvelope.kt new file mode 100644 index 00000000..f15099bc --- /dev/null +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunEnvelope.kt @@ -0,0 +1,229 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import org.jetbrains.annotations.ApiStatus + +/** + * The digests that say what a run ran against. + * + * All three are opaque to DICE: it compares them and stores them and reads nothing out of them. + * They are what makes "did the output change because the prompt changed?" answerable without + * storing the prompt. A host that changes a template and forgets to change its fingerprint gets + * runs it cannot tell apart, which is the host's contract to keep. + * + * Storing digests instead of the material is deliberate: a prompt template holds instructions and + * often examples, and examples are where real content ends up. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + * + * @property promptTemplateFingerprint Digest of the prompt or template the run used + * @property schemaFingerprint Digest of the output schema the run asked the model to satisfy + * @property metamodelFingerprint Digest of the metamodel version in force + */ +@ApiStatus.Experimental +data class ExtractionRunFingerprints @JvmOverloads constructor( + val promptTemplateFingerprint: String? = null, + val schemaFingerprint: String? = null, + val metamodelFingerprint: String? = null, +) { + + init { + requireBoundedIdentifier(promptTemplateFingerprint, "promptTemplateFingerprint") + requireBoundedIdentifier(schemaFingerprint, "schemaFingerprint") + requireBoundedIdentifier(metamodelFingerprint, "metamodelFingerprint") + } + + companion object { + + /** Java-friendly factory. */ + @JvmStatic + @JvmOverloads + fun of( + promptTemplateFingerprint: String? = null, + schemaFingerprint: String? = null, + metamodelFingerprint: String? = null, + ): ExtractionRunFingerprints = ExtractionRunFingerprints( + promptTemplateFingerprint = promptTemplateFingerprint, + schemaFingerprint = schemaFingerprint, + metamodelFingerprint = metamodelFingerprint, + ) + } +} + +/** + * What code ran the run, and where. + * + * The version fields are what separate "the extractor changed" from "the model changed" when + * output quality moves. OpenLineage's processing-engine facet carries the same pair of a name and + * a version for the same reason. + * + * [hostApplication] names the application embedding DICE, not a machine. A hostname would be a + * direct identifier of infrastructure and belongs in the host's own telemetry. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + * + * @property extractor Which extractor implementation ran + * @property extractorVersion Its version + * @property hostApplication The application that embeds DICE + * @property runtime The runtime the run executed on, such as a DICE release or a service name + * @property runtimeVersion Its version + */ +@ApiStatus.Experimental +data class ExtractionRuntimeIdentity @JvmOverloads constructor( + val extractor: String? = null, + val extractorVersion: String? = null, + val hostApplication: String? = null, + val runtime: String? = null, + val runtimeVersion: String? = null, +) { + + init { + requireBoundedIdentifier(extractor, "extractor") + requireBoundedIdentifier(extractorVersion, "extractorVersion") + requireBoundedIdentifier(hostApplication, "hostApplication") + requireBoundedIdentifier(runtime, "runtime") + requireBoundedIdentifier(runtimeVersion, "runtimeVersion") + } + + companion object { + + /** Java-friendly factory. */ + @JvmStatic + @JvmOverloads + fun of( + extractor: String? = null, + extractorVersion: String? = null, + hostApplication: String? = null, + runtime: String? = null, + runtimeVersion: String? = null, + ): ExtractionRuntimeIdentity = ExtractionRuntimeIdentity( + extractor = extractor, + extractorVersion = extractorVersion, + hostApplication = hostApplication, + runtime = runtime, + runtimeVersion = runtimeVersion, + ) + } +} + +/** + * How much a run got through. + * + * Counts are what a run page sorts and filters on, so they sit on the header rather than being + * derived by counting rows. They are recorded by whoever ran the extraction; nothing here + * cross-checks one against another, because a run that extracted 40 propositions and persisted 12 + * is a real and interesting state, not a contradiction. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + * + * @property sourcesRead How many source revisions the run read + * @property chunksProcessed How many chunks reached extraction + * @property propositionsExtracted How many propositions the model produced + * @property propositionsPersisted How many were stored + * @property propositionsRejected How many were dropped by a gate or a filter + * @property entitiesResolved How many entity mentions were resolved + */ +@ApiStatus.Experimental +data class ExtractionRunCounts @JvmOverloads constructor( + val sourcesRead: Int = 0, + val chunksProcessed: Int = 0, + val propositionsExtracted: Int = 0, + val propositionsPersisted: Int = 0, + val propositionsRejected: Int = 0, + val entitiesResolved: Int = 0, +) { + + init { + requireNonNegative(sourcesRead, "sourcesRead") + requireNonNegative(chunksProcessed, "chunksProcessed") + requireNonNegative(propositionsExtracted, "propositionsExtracted") + requireNonNegative(propositionsPersisted, "propositionsPersisted") + requireNonNegative(propositionsRejected, "propositionsRejected") + requireNonNegative(entitiesResolved, "entitiesResolved") + } + + companion object { + + /** Java-friendly factory. */ + @JvmStatic + @JvmOverloads + fun of( + sourcesRead: Int = 0, + chunksProcessed: Int = 0, + propositionsExtracted: Int = 0, + propositionsPersisted: Int = 0, + propositionsRejected: Int = 0, + entitiesResolved: Int = 0, + ): ExtractionRunCounts = ExtractionRunCounts( + sourcesRead = sourcesRead, + chunksProcessed = chunksProcessed, + propositionsExtracted = propositionsExtracted, + propositionsPersisted = propositionsPersisted, + propositionsRejected = propositionsRejected, + entitiesResolved = entitiesResolved, + ) + } +} + +/** + * The five pseudonymous references a run carries about whose work it was. + * + * Grouped together because they share one contract, stated on [ExtractionOpaqueRef]: bounded, + * host-minted, no direct identifiers, nothing dereferenceable. Grouping them also keeps them + * findable — a reader looking for "what does a run know about the user?" gets one type with five + * fields and an answer. + * + * Every field is optional. A run started by a scheduled job has an actor reference and no session; + * a run replaying archived material may have neither. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + * + * @property actor Who the run acted for + * @property request The inbound request that caused it + * @property session The conversation or session the material came from + * @property personalization The personalization state in force + * @property deployment The deployment it executed in + */ +@ApiStatus.Experimental +data class ExtractionRunSubjectRefs @JvmOverloads constructor( + val actor: ExtractionActorRef? = null, + val request: ExtractionRequestRef? = null, + val session: ExtractionSessionRef? = null, + val personalization: ExtractionPersonalizationRef? = null, + val deployment: ExtractionDeploymentRef? = null, +) { + + companion object { + + /** Java-friendly factory. */ + @JvmStatic + @JvmOverloads + fun of( + actor: ExtractionActorRef? = null, + request: ExtractionRequestRef? = null, + session: ExtractionSessionRef? = null, + personalization: ExtractionPersonalizationRef? = null, + deployment: ExtractionDeploymentRef? = null, + ): ExtractionRunSubjectRefs = ExtractionRunSubjectRefs( + actor = actor, + request = request, + session = session, + personalization = personalization, + deployment = deployment, + ) + } +} diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLimits.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLimits.kt new file mode 100644 index 00000000..14cbc98d --- /dev/null +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLimits.kt @@ -0,0 +1,125 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import org.jetbrains.annotations.ApiStatus + +/** + * Every bound an extraction run obeys, in one place. + * + * One rule covers the whole run model: each bound is a named constant here, the check runs in the + * `init` block of the type that owns the value, and anything over the bound is rejected at + * construction. Truncating an identifier would be worse than rejecting it — a shortened id is a + * different id, and a store would then key rows on a value the caller never minted. + * + * A failure detail is the one exception, and only on the way in. It is the model's only free-text + * field, so [ExtractionFailure.of] and [ExtractionFailure.fromThrowable] shorten it to + * [MAX_FAILURE_DETAIL_LENGTH] before construction; keeping a clipped failure record beats losing + * the failure. The constructor still rejects a longer one. + * + * Lengths count UTF-16 chars (`String.length`), so a 256-char identifier can be around 1 KB of + * UTF-8. The bound is there to keep a run header finite. + * + * `SourceRevisionRef` predates this rule and validates non-blank only, so [ExtractionRun] applies + * [MAX_SOURCE_KEY_LENGTH] and [MAX_IDENTIFIER_LENGTH] to the revisions it stores. Moving those + * checks into `SourceRevisionRef` is a follow-up in the module that owns it. + * + * One string a run stores is outside the rule and stays outside it: `ContextId.value`, which + * `ExtractionRun` holds as its tenant and validates non-blank only. `ContextId` is a DICE-wide + * type owned by the agent framework, so bounding it is not this model's call. It matters because + * the tenant is half of the store key `ExtractionRunKey`, so the run store's key length is bounded + * on one side only; whoever sizes that key's index decides what to do about the other side. + */ +@ApiStatus.Experimental +object ExtractionRunLimits { + + /** + * Longest host-minted identifier a run stores: opaque reference tokens, fingerprints, model + * and role names, service names, provider response ids, runtime identifiers. A uuid, a ULID, + * a sha-256 hex digest, or a host correlation id all fit with room to spare. + */ + const val MAX_IDENTIFIER_LENGTH: Int = 256 + + /** + * Longest source key a run stores. Source keys come out of `SourceLocator.key()` and can + * legitimately hold a long URL, so they get more room than an identifier a host mints itself. + */ + const val MAX_SOURCE_KEY_LENGTH: Int = 1024 + + /** + * Longest failure detail a run stores. Long enough for a classified one-line explanation, + * short enough that a run with the full [MAX_FAILURES] of them stays small. + */ + const val MAX_FAILURE_DETAIL_LENGTH: Int = 512 + + /** Most source revisions one run may record. */ + const val MAX_SOURCE_REVISIONS: Int = 256 + + /** Most invocation records one run may record, across every invocation and every attempt. */ + const val MAX_INVOCATIONS: Int = 1024 + + /** + * Most failure records one run may record. A run that fails this many times has a systemic + * problem, and the hundredth message says nothing the first ten did not. + */ + const val MAX_FAILURES: Int = 64 +} + +/** + * Checks a host-minted identifier that may be absent. + * + * Error messages name the field and the length and never quote the value, because an + * `IllegalArgumentException` propagates into logs and a rejected value is exactly the one nobody + * vouched for. + */ +internal fun requireBoundedIdentifier(value: String?, field: String): String? { + if (value == null) return null + require(value.isNotBlank()) { "$field must not be blank when present" } + require(value.length <= ExtractionRunLimits.MAX_IDENTIFIER_LENGTH) { + "$field must be at most ${ExtractionRunLimits.MAX_IDENTIFIER_LENGTH} characters, was ${value.length}" + } + return value +} + +/** Checks a count that may be absent and can never be negative. */ +internal fun requireNonNegative(value: Int?, field: String): Int? { + if (value == null) return null + require(value >= 0) { "$field must not be negative, was $value" } + return value +} + +/** Checks a double that may be absent and must be a real number. */ +internal fun requireFinite(value: Double?, field: String): Double? { + if (value == null) return null + require(value.isFinite()) { "$field must be a finite number, was $value" } + return value +} + +/** Checks a double that may be absent, must be a real number, and must not fall below [min]. */ +internal fun requireAtLeast(value: Double?, field: String, min: Double): Double? { + requireFinite(value, field) + if (value == null) return null + require(value >= min) { "$field must be at least $min, was $value" } + return value +} + +/** Checks a double that may be absent, must be a real number, and must sit within [min]..[max]. */ +internal fun requireInRange(value: Double?, field: String, min: Double, max: Double): Double? { + requireAtLeast(value, field, min) + if (value == null) return null + require(value <= max) { "$field must be at most $max, was $value" } + return value +} diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLineage.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLineage.kt new file mode 100644 index 00000000..de51e2bd --- /dev/null +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLineage.kt @@ -0,0 +1,120 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import org.jetbrains.annotations.ApiStatus + +/** + * Where a run sits among the runs around it: its own reference, its parent, its root, what it + * supersedes, and which pass it is. + * + * Parent and supersession are two different axes and stay separate fields. A parent is the run + * this one continues from — a later pass reading the entities its parent resolved. A superseded + * run is one this one replaces — a re-extraction after the prompt changed. A run can have one of + * each, both, or neither. + * + * **The root reference is denormalized on purpose.** It is set once when the lineage is minted: + * a run with no parent is its own root, and a run with a parent takes its parent's root. That + * makes "everything in this lineage" a single indexed read on one property, instead of walking + * the parent chain a hop at a time. OpenLineage's `ParentRunFacet` does the same thing — it + * carries an optional `root` alongside the immediate parent, so consumers do not have to walk — + * and deep pass-and-retry chains are exactly where walking hurts. + * + * The root only stays true if it is never recomputed later, so it is fixed at construction. Use + * [root] and [childOf] and the arithmetic is done for you; the constructor rejects the + * combinations that would make the field a lie. + * + * **What is checked here, and what is not.** The constructor rejects self-reference on both axes + * and rejects a root that contradicts the presence of a parent. It cannot see a cycle of length + * two or more, because a value type holds one run and cycle detection needs the other runs. The + * store that walks these chains is where bounded, cycle-safe traversal lives. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + * + * @property runRef This run + * @property rootRunRef The oldest run in this lineage, which is [runRef] itself when there is no + * parent + * @property parentRunRef The run this one continues from, or null + * @property supersedesRunRef The run this one replaces, or null + * @property passIndex Which pass over the material this is, counting from zero + */ +@ApiStatus.Experimental +data class ExtractionRunLineage @JvmOverloads constructor( + val runRef: ExtractionRunRef, + val rootRunRef: ExtractionRunRef, + val parentRunRef: ExtractionRunRef? = null, + val supersedesRunRef: ExtractionRunRef? = null, + val passIndex: Int = 0, +) { + + init { + require(passIndex >= 0) { "passIndex must not be negative, was $passIndex" } + require(parentRunRef != runRef) { "a run cannot be its own parent" } + require(supersedesRunRef != runRef) { "a run cannot supersede itself" } + if (parentRunRef == null) { + require(rootRunRef == runRef) { "a run with no parent is its own root" } + } else { + require(rootRunRef != runRef) { "a run with a parent takes its parent's root, so it is not its own root" } + } + } + + /** True when this run starts a lineage. */ + val isRoot: Boolean + get() = parentRunRef == null + + companion object { + + /** + * Mints the lineage of a run that starts one: its own root, no parent. + * + * A first extraction of some material takes this. So does a re-extraction that replaces + * an earlier run without continuing it — pass [supersedesRunRef] and the supersession is + * recorded without making the replaced run a parent. + */ + @JvmStatic + @JvmOverloads + fun root( + runRef: ExtractionRunRef, + supersedesRunRef: ExtractionRunRef? = null, + passIndex: Int = 0, + ): ExtractionRunLineage = ExtractionRunLineage( + runRef = runRef, + rootRunRef = runRef, + parentRunRef = null, + supersedesRunRef = supersedesRunRef, + passIndex = passIndex, + ) + + /** + * Mints the lineage of a run that continues [parent], carrying the parent's root forward + * and defaulting to the next pass index. + */ + @JvmStatic + @JvmOverloads + fun childOf( + runRef: ExtractionRunRef, + parent: ExtractionRunLineage, + supersedesRunRef: ExtractionRunRef? = null, + passIndex: Int = parent.passIndex + 1, + ): ExtractionRunLineage = ExtractionRunLineage( + runRef = runRef, + rootRunRef = parent.rootRunRef, + parentRunRef = parent.runRef, + supersedesRunRef = supersedesRunRef, + passIndex = passIndex, + ) + } +} diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunRef.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunRef.kt new file mode 100644 index 00000000..37ebb426 --- /dev/null +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunRef.kt @@ -0,0 +1,68 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import org.jetbrains.annotations.ApiStatus + +/** + * Names one extraction run. + * + * This is identity and nothing else. It holds no timing, no status, no counts, no lineage — + * just the id, so a caller can say "this analysis belongs to that run" without DICE having + * anywhere to store a run yet. Durable extraction runs arrive with DICE #67 and will be keyed + * by ([com.embabel.agent.core.ContextId], `ExtractionRunRef`); shipping the reference first + * means the entry points and the run model meet at an opaque string rather than at a type one + * of them has to import from the other's release. + * + * The id is opaque. DICE compares it and carries it and parses nothing out of it. It also never + * mints one: a run is something the host (or, later, DICE's own run coordinator) starts. Nothing + * here checks that the run exists, because there is nowhere yet to check against — carrying a + * reference is always allowed. What a store does with a reference to a run it has never seen is + * DICE #67's to decide, and this type makes no promise about it either way. + * + * Run identity is deliberately not part of source-provenance equality. Two runs over the same + * material still produce one piece of source evidence; what differs is which runs are + * attributed to it. + * + * A run reference is not an authorization token and must not carry a secret, a direct + * identifier, or anything a reader could dereference into personal data. Hosts mint it. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + * + * @property runId Host-minted opaque identifier for the run + */ +@ApiStatus.Experimental +data class ExtractionRunRef( + val runId: String, +) { + + init { + require(runId.isNotBlank()) { "runId must not be blank" } + require(runId.length <= MAX_RUN_ID_LENGTH) { + "runId must be at most $MAX_RUN_ID_LENGTH characters, was ${runId.length}" + } + } + + companion object { + + /** + * Longest run id DICE accepts. #67 keys stored runs on this string and indexes it, so + * an unbounded id would become an unbounded key. A uuid, a ULID, or a host's own + * correlation id all fit with room to spare. + */ + const val MAX_RUN_ID_LENGTH: Int = 256 + } +} diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunStatus.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunStatus.kt new file mode 100644 index 00000000..2ec2bb4f --- /dev/null +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunStatus.kt @@ -0,0 +1,73 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import org.jetbrains.annotations.ApiStatus + +/** + * Where an extraction run stands. + * + * These are the four values, and only the values. Which transitions are legal, which are + * compare-and-set, and what a store does with a repeated terminal write all belong to the run + * store contract that lands with DICE #67's next slice. Nothing here encodes a transition rule, + * so the state machine has one place to define them. + * + * MLflow's run status carries five values — `RUNNING`, `SCHEDULED`, `FINISHED`, `FAILED`, + * `KILLED`. DICE has four for two reasons. There is no scheduler, so nothing can observe a run + * between "requested" and "started" and `SCHEDULED` would never be written. And a run stopped + * from outside is recorded as [CANCELLED], because the fact an operator or an audit cares about + * is that the run stopped short of its products, which is the same fact whichever side pressed + * stop. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + */ +@ApiStatus.Experimental +enum class ExtractionRunStatus { + + /** + * The run started and has not reached a terminal state. A run whose process died mid-way + * stays here until something moves it, which is what makes it retryable. + */ + RUNNING, + + /** + * Every product the run's request called for is either durably persisted or terminally + * disposed. + * + * That is the definition the store contract enforces, and it is stated here so the meaning + * travels with the value. The consequence worth knowing at the model layer: COMPLETED is + * written after persistence, never before, so a run whose persistence never finished stays + * [RUNNING] rather than claiming products it does not have. A run with zero products + * terminalizes COMPLETED — vacuously, since there was nothing left to persist. + */ + COMPLETED, + + /** The run stopped on an error it could not get past. Its recorded failures say which. */ + FAILED, + + /** + * The run stopped before finishing, by request or by external termination. This is also the + * abandonment path for a partially successful run nobody intends to finish: its outstanding + * products stay outstanding behind it, and recovery goes through a new run linked by parent + * or superseded reference. + */ + CANCELLED, + ; + + /** True for the three states a run does not leave. */ + val isTerminal: Boolean + get() = this != RUNNING +} diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionInvocationIdentityTest.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionInvocationIdentityTest.kt new file mode 100644 index 00000000..712fc2aa --- /dev/null +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionInvocationIdentityTest.kt @@ -0,0 +1,278 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import org.assertj.core.api.Assertions.assertThat +import org.assertj.core.api.Assertions.assertThatIllegalArgumentException +import org.junit.jupiter.api.Test +import java.lang.reflect.Modifier + +/** + * Invocation identity is allocated when the call plan is laid out and never derived from the order + * answers come back. These tests pin both halves: the identities a plan hands out survive + * out-of-order completion and retries, and the types offer no way to mint one from a position in a + * result list. + */ +class ExtractionInvocationIdentityTest { + + private val startedAt = ExtractionRunFixtures.STARTED_AT + + @Test + fun `a plan allocates every identity before anything is dispatched`() { + val plan = ExtractionInvocationRecord.plan(4) + + assertThat(plan.map { it.invocationIndex }).containsExactly(0, 1, 2, 3) + assertThat(plan.map { it.attempt }).containsExactly(1, 1, 1, 1) + assertThat(plan.map { it.outcome }) + .allMatch { it == ExtractionInvocationOutcome.IN_FLIGHT } + // Nothing has been dispatched, so there is no timing and nothing observed. + assertThat(plan.map { it.startedAt }).allMatch { it == null } + assertThat(plan.map { it.usage }).allMatch { it == null } + assertThat(plan.map { it.providerResponse }).allMatch { it == null } + } + + @Test + fun `completion order does not touch identity`() { + val plan = ExtractionInvocationRecord.plan(4) + + // Calls come back in the order 2, 0, 3, 1. Each answer writes into the identity its call + // already had. + val completed = listOf(2, 0, 3, 1).mapIndexed { arrivalPosition, planIndex -> + plan[planIndex].copy( + outcome = ExtractionInvocationOutcome.SUCCEEDED, + startedAt = startedAt, + finishedAt = startedAt.plusSeconds(arrivalPosition + 1L), + ) + } + + assertThat(completed.map { it.invocationIndex }).containsExactly(2, 0, 3, 1) + assertThat(completed.map { it.id }).containsExactlyElementsOf( + listOf(2, 0, 3, 1).map { ExtractionInvocationId.planned(it) }, + ) + + val run = runWith(completed) + + // Stored in arrival order, read back in plan order. + assertThat(run.invocations.map { it.invocationIndex }).containsExactly(2, 0, 3, 1) + assertThat(run.invocationsInPlanOrder().map { it.invocationIndex }).containsExactly(0, 1, 2, 3) + } + + @Test + fun `an attempt numbers a retry of the same call`() { + val first = ExtractionInvocationRecord.planned(2).copy( + outcome = ExtractionInvocationOutcome.FAILED, + configuredService = "service-alpha", + startedAt = startedAt, + finishedAt = startedAt.plusSeconds(3), + usage = ExtractionModelUsage(inputTokens = 100), + ) + + val second = first.retry() + val third = second.retry() + + assertThat(listOf(first, second, third).map { it.invocationIndex }).containsExactly(2, 2, 2) + assertThat(listOf(first, second, third).map { it.attempt }).containsExactly(1, 2, 3) + // A retry carries the identity forward and nothing else: the observations belonged to the + // attempt that just failed. + assertThat(second.outcome).isEqualTo(ExtractionInvocationOutcome.IN_FLIGHT) + assertThat(second.configuredService).isNull() + assertThat(second.startedAt).isNull() + assertThat(second.finishedAt).isNull() + assertThat(second.usage).isNull() + } + + @Test + fun `attempts of one call sort under it, whatever order they were recorded in`() { + val run = runWith( + listOf( + ExtractionInvocationRecord(ExtractionInvocationId(1, 2)), + ExtractionInvocationRecord(ExtractionInvocationId(0, 1)), + ExtractionInvocationRecord(ExtractionInvocationId(1, 1)), + ExtractionInvocationRecord(ExtractionInvocationId(1, 3)), + ), + ) + + assertThat(run.invocationsInPlanOrder().map { "${it.invocationIndex}/${it.attempt}" }) + .containsExactly("0/1", "1/1", "1/2", "1/3") + assertThat(run.attemptsOf(1).map { it.attempt }).containsExactly(1, 2, 3) + assertThat(run.attemptsOf(7)).isEmpty() + } + + @Test + fun `a run rejects two records with the same identity`() { + assertThatIllegalArgumentException().isThrownBy { + runWith( + listOf( + ExtractionInvocationRecord(ExtractionInvocationId(0, 1)), + ExtractionInvocationRecord(ExtractionInvocationId(0, 1)), + ), + ) + }.withMessageContaining("distinct") + } + + @Test + fun `identities are validated at the edges`() { + assertThatIllegalArgumentException().isThrownBy { ExtractionInvocationId(-1, 1) } + .withMessageContaining("invocationIndex") + assertThatIllegalArgumentException().isThrownBy { ExtractionInvocationId(0, 0) } + .withMessageContaining("attempt") + assertThatIllegalArgumentException().isThrownBy { ExtractionInvocationRecord.plan(-1) } + .withMessageContaining("count") + assertThat(ExtractionInvocationRecord.plan(0)).isEmpty() + } + + @Test + fun `a plan is bounded before it is allocated`() { + // A plan size derived from chunking a large document can be enormous. The bound is checked + // on the count, so an over-limit plan costs nothing rather than building the list first and + // failing at the run. + val atLimit = ExtractionInvocationRecord.plan(ExtractionRunLimits.MAX_INVOCATIONS) + + assertThat(atLimit).hasSize(ExtractionRunLimits.MAX_INVOCATIONS) + assertThat(atLimit.last().invocationIndex).isEqualTo(ExtractionRunLimits.MAX_INVOCATIONS - 1) + + assertThatIllegalArgumentException() + .isThrownBy { ExtractionInvocationRecord.plan(ExtractionRunLimits.MAX_INVOCATIONS + 1) } + .withMessageContaining("call plan may hold at most") + assertThatIllegalArgumentException() + .isThrownBy { ExtractionInvocationRecord.plan(Int.MAX_VALUE) } + .withMessageContaining("call plan may hold at most") + } + + @Test + fun `timing is an observation and may be absent on a terminal record`() { + // Deliberate: a recorded outcome with no clock means the timing was not observed, not that + // the call did not run. Requiring it would push callers to invent a duration. + val succeededWithoutTiming = ExtractionInvocationRecord( + id = ExtractionInvocationId.planned(0), + outcome = ExtractionInvocationOutcome.SUCCEEDED, + configuredService = "service-alpha", + usage = ExtractionModelUsage(inputTokens = 100), + ) + + assertThat(succeededWithoutTiming.startedAt).isNull() + assertThat(succeededWithoutTiming.finishedAt).isNull() + assertThat(succeededWithoutTiming.usage?.inputTokens).isEqualTo(100) + + // A started-but-untimed finish is recordable too, which is the half a dispatcher knows. + val startedNotTimed = succeededWithoutTiming.copy(startedAt = startedAt) + assertThat(startedNotTimed.finishedAt).isNull() + } + + @Test + fun `an in-flight attempt cannot have finished and a finish cannot precede a start`() { + assertThatIllegalArgumentException().isThrownBy { + ExtractionInvocationRecord( + id = ExtractionInvocationId.planned(0), + startedAt = startedAt, + finishedAt = startedAt.plusSeconds(1), + ) + }.withMessageContaining("IN_FLIGHT") + + assertThatIllegalArgumentException().isThrownBy { + ExtractionInvocationRecord( + id = ExtractionInvocationId.planned(0), + outcome = ExtractionInvocationOutcome.SUCCEEDED, + startedAt = startedAt, + finishedAt = startedAt.minusSeconds(1), + ) + }.withMessageContaining("finishedAt") + + assertThatIllegalArgumentException().isThrownBy { + ExtractionInvocationRecord( + id = ExtractionInvocationId.planned(0), + outcome = ExtractionInvocationOutcome.SUCCEEDED, + finishedAt = startedAt, + ) + }.withMessageContaining("without having started") + } + + @Test + fun `an attempt cancelled before dispatch is recordable`() { + val cancelled = ExtractionInvocationRecord( + id = ExtractionInvocationId.planned(3), + outcome = ExtractionInvocationOutcome.CANCELLED, + ) + + assertThat(cancelled.startedAt).isNull() + assertThat(cancelled.invocationIndex).isEqualTo(3) + } + + @Test + fun `nothing can mint an identity out of a completion position`() { + // Every way to build a record demands an identity, and the only way to build an identity + // is from a plan ordinal. There is no factory taking a result-list position, and this + // asserts that rather than describing it. + val recordFactories = ExtractionInvocationRecord::class.java.declaredMethods + .filter { Modifier.isStatic(it.modifiers) && Modifier.isPublic(it.modifiers) } + .filterNot { it.isSynthetic || it.name.contains('$') } + .filter { it.returnType == ExtractionInvocationRecord::class.java } + assertThat(recordFactories.map { it.name }).containsExactly("planned") + assertThat(recordFactories.single().parameterTypes).containsExactly(Int::class.javaPrimitiveType) + + val recordConstructors = ExtractionInvocationRecord::class.java.constructors + assertThat(recordConstructors).allSatisfy { constructor -> + assertThat(constructor.parameterTypes.first()).isEqualTo(ExtractionInvocationId::class.java) + } + + val idFactories = ExtractionInvocationId::class.java.declaredMethods + .filter { Modifier.isStatic(it.modifiers) && Modifier.isPublic(it.modifiers) } + .filterNot { it.isSynthetic || it.name.contains('$') } + .filter { it.returnType == ExtractionInvocationId::class.java } + assertThat(idFactories.map { it.name }).containsExactly("planned") + } + + @Test + fun `an invocation record cannot hold a requested configuration`() { + // The separation is structural. No observed type has a field of the requested type, so a + // mapper cannot fold one into the other and a reader cannot mistake the two. + val observedTypes = listOf( + ExtractionInvocationRecord::class.java, + ExtractionModelUsage::class.java, + ExtractionProviderResponseFacts::class.java, + ) + observedTypes.forEach { type -> + assertThat(type.declaredFields.map { it.type }) + .doesNotContain(ExtractionRequestedModelConfig::class.java) + } + } + + @Test + fun `requested and observed model facts share no property name`() { + // Different names on purpose: `requestedModel` is what was asked for and `responseModel` + // is what the provider said answered. A shared name is how one silently becomes the other. + val requested = propertyNames(ExtractionRequestedModelConfig::class.java) + val observed = propertyNames(ExtractionProviderResponseFacts::class.java) + + propertyNames(ExtractionModelUsage::class.java) + + assertThat(requested).isNotEmpty() + assertThat(observed).isNotEmpty() + assertThat(requested.intersect(observed)).isEmpty() + } + + private fun propertyNames(type: Class<*>): Set = type.declaredFields + .filterNot { it.isSynthetic || Modifier.isStatic(it.modifiers) } + .map { it.name } + .toSet() + + private fun runWith(invocations: List): ExtractionRun = ExtractionRun( + contextId = ExtractionRunFixtures.CONTEXT, + lineage = ExtractionRunLineage.root(ExtractionRunFixtures.RUN), + status = ExtractionRunStatus.RUNNING, + startedAt = startedAt, + invocations = invocations, + ) +} diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunContractTest.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunContractTest.kt new file mode 100644 index 00000000..c1fb9a09 --- /dev/null +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunContractTest.kt @@ -0,0 +1,432 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import com.embabel.agent.core.ContextId +import com.embabel.dice.proposition.extraction.ExtractionRunFixtures.CONTEXT +import com.embabel.dice.proposition.extraction.ExtractionRunFixtures.OTHER_CONTEXT +import com.embabel.dice.proposition.extraction.ExtractionRunFixtures.REVISION_ONE +import com.embabel.dice.proposition.extraction.ExtractionRunFixtures.REVISION_TWO +import com.embabel.dice.proposition.extraction.ExtractionRunFixtures.RUN +import com.embabel.dice.proposition.extraction.ExtractionRunFixtures.STARTED_AT +import com.embabel.dice.provenance.SourceRevisionRef +import org.assertj.core.api.Assertions.assertThat +import org.assertj.core.api.Assertions.assertThatIllegalArgumentException +import org.junit.jupiter.api.Test +import org.junit.jupiter.api.assertThrows + +/** + * What an [ExtractionRun] accepts, what it rejects, what it copies, and when two of them are the + * same run. + */ +class ExtractionRunContractTest { + + @Test + fun `a started run needs a tenant, a lineage, a status and a start`() { + val run = ExtractionRunFixtures.startedRun() + + assertThat(run.contextId).isEqualTo(CONTEXT) + assertThat(run.ref).isEqualTo(RUN) + assertThat(run.status).isEqualTo(ExtractionRunStatus.RUNNING) + assertThat(run.startedAt).isEqualTo(STARTED_AT) + assertThat(run.finishedAt).isNull() + // Everything else defaults to absent or empty, so an empty run is a run. + assertThat(run.sourceRevisions).isEmpty() + assertThat(run.invocations).isEmpty() + assertThat(run.failures).isEmpty() + assertThat(run.profile).isNull() + assertThat(run.requestedModel).isNull() + assertThat(run.replayFidelity).isEqualTo(ExtractionReplayFidelity.NONE) + assertThat(run.counts).isEqualTo(ExtractionRunCounts()) + } + + @Test + fun `a populated run reads back every field it was given`() { + val run = ExtractionRunFixtures.populatedRun() + + assertThat(run.status).isEqualTo(ExtractionRunStatus.FAILED) + assertThat(run.status.isTerminal).isTrue() + assertThat(run.finishedAt).isEqualTo(ExtractionRunFixtures.FINISHED_AT) + assertThat(run.profile).isEqualTo(ExtractionContentProfileRef("house-style", "v3")) + assertThat(run.sourceRevisions).containsExactly(REVISION_ONE, REVISION_TWO) + assertThat(run.fingerprints.schemaFingerprint).isEqualTo("sha256:a7c40e19") + assertThat(run.runtime.extractor).isEqualTo("LlmPropositionExtractor") + assertThat(run.requestedModel?.temperature).isEqualTo(0.2) + assertThat(run.subjectRefs.actor).isEqualTo(ExtractionActorRef("actor:7f19aa02")) + assertThat(run.experimentRef).isEqualTo(ExtractionExperimentRef("exp:prompt-v3")) + assertThat(run.cohortRef).isEqualTo(ExtractionCohortRef("cohort:treatment")) + assertThat(run.replayFidelity).isEqualTo(ExtractionReplayFidelity.APPROXIMATE) + assertThat(run.counts.propositionsPersisted).isEqualTo(11) + assertThat(run.invocations).hasSize(2) + assertThat(run.failures).hasSize(1) + assertThat(run.failures.single().code).isEqualTo(ExtractionFailureCode.DECODE_FAILED) + assertThat(run.failures.single().invocation).isEqualTo(ExtractionInvocationId(1, 2)) + } + + @Test + fun `a finish cannot precede a start`() { + assertThatIllegalArgumentException().isThrownBy { + ExtractionRun( + contextId = CONTEXT, + lineage = ExtractionRunLineage.root(RUN), + status = ExtractionRunStatus.COMPLETED, + startedAt = STARTED_AT, + finishedAt = STARTED_AT.minusSeconds(1), + ) + }.withMessageContaining("finishedAt") + + // The same instant is fine: a run that did nothing can finish in the tick it started. + assertThat( + ExtractionRun( + contextId = CONTEXT, + lineage = ExtractionRunLineage.root(RUN), + status = ExtractionRunStatus.COMPLETED, + startedAt = STARTED_AT, + finishedAt = STARTED_AT, + ).finishedAt, + ).isEqualTo(STARTED_AT) + } + + @Test + fun `which status may sit with which timing is left to the lifecycle`() { + // Deliberate: the run store contract owns the transitions and the definition of COMPLETED, + // so this type does not half-encode them. A terminal status with no finish time is + // constructible here and is the state machine's to reject. + val terminalWithoutFinish = ExtractionRun( + contextId = CONTEXT, + lineage = ExtractionRunLineage.root(RUN), + status = ExtractionRunStatus.CANCELLED, + startedAt = STARTED_AT, + ) + + assertThat(terminalWithoutFinish.finishedAt).isNull() + assertThat(terminalWithoutFinish.status.isTerminal).isTrue() + assertThat(ExtractionRunStatus.entries.map { it.name }) + .containsExactly("RUNNING", "COMPLETED", "FAILED", "CANCELLED") + assertThat(ExtractionRunStatus.entries.filter { it.isTerminal }) + .containsExactly( + ExtractionRunStatus.COMPLETED, + ExtractionRunStatus.FAILED, + ExtractionRunStatus.CANCELLED, + ) + } + + @Test + fun `source revisions are ordered and distinct`() { + val run = runWith(sourceRevisions = listOf(REVISION_TWO, REVISION_ONE)) + + assertThat(run.sourceRevisions).containsExactly(REVISION_TWO, REVISION_ONE) + + assertThatIllegalArgumentException().isThrownBy { + runWith(sourceRevisions = listOf(REVISION_ONE, REVISION_ONE)) + }.withMessageContaining("distinct") + } + + @Test + fun `every collection the run stores is bounded`() { + assertThatIllegalArgumentException().isThrownBy { + runWith( + sourceRevisions = (0..ExtractionRunLimits.MAX_SOURCE_REVISIONS) + .map { SourceRevisionRef("uri:doc-$it", "rev-1") }, + ) + }.withMessageContaining("source revisions") + + assertThatIllegalArgumentException().isThrownBy { + runWith( + failures = (0..ExtractionRunLimits.MAX_FAILURES) + .map { ExtractionFailure(ExtractionFailureCode.INTERNAL, "failure $it", STARTED_AT) }, + ) + }.withMessageContaining("failures") + + assertThatIllegalArgumentException().isThrownBy { + runWith( + invocations = (0..ExtractionRunLimits.MAX_INVOCATIONS) + .map { ExtractionInvocationRecord.planned(it) }, + ) + }.withMessageContaining("invocation records") + } + + @Test + fun `the run applies the cap rule to source revisions its own type does not check`() { + // SourceRevisionRef predates the rule and validates non-blank only, so the bound is applied + // where the run stores it. + assertThatIllegalArgumentException().isThrownBy { + runWith( + sourceRevisions = listOf( + SourceRevisionRef("u".repeat(ExtractionRunLimits.MAX_SOURCE_KEY_LENGTH + 1), "rev-1"), + ), + ) + }.withMessageContaining("sourceKey") + + assertThatIllegalArgumentException().isThrownBy { + runWith( + sourceRevisions = listOf( + SourceRevisionRef("uri:doc-a", "r".repeat(ExtractionRunLimits.MAX_IDENTIFIER_LENGTH + 1)), + ), + ) + }.withMessageContaining("sourceRevision") + + // A long URL as a source key is exactly why source keys get more room than identifiers. + val longUrlKey = "https://example.test/" + "segment/".repeat(60) + assertThat(longUrlKey.length).isGreaterThan(ExtractionRunLimits.MAX_IDENTIFIER_LENGTH) + assertThat(runWith(sourceRevisions = listOf(SourceRevisionRef(longUrlKey, "rev-1"))).sourceRevisions) + .hasSize(1) + } + + @Test + fun `a failure cannot name an attempt the run has no record of`() { + val plan = ExtractionInvocationRecord.plan(2) + + // A failure tied to an attempt the run does record is the normal case. + assertThat( + runWith( + invocations = plan, + failures = listOf( + ExtractionFailure( + code = ExtractionFailureCode.MODEL_TIMEOUT, + at = STARTED_AT, + invocation = ExtractionInvocationId.planned(1), + ), + ), + ).failures, + ).hasSize(1) + + // An index outside the plan is a dangling audit reference: it reads as evidence about a + // call and nothing can join it to one. + assertThatIllegalArgumentException().isThrownBy { + runWith( + invocations = plan, + failures = listOf( + ExtractionFailure( + code = ExtractionFailureCode.MODEL_TIMEOUT, + at = STARTED_AT, + invocation = ExtractionInvocationId.planned(7), + ), + ), + ) + }.withMessageContaining("invocation 7 attempt 1") + + // So is the right call at an attempt that was never made. + assertThatIllegalArgumentException().isThrownBy { + runWith( + invocations = plan, + failures = listOf( + ExtractionFailure( + code = ExtractionFailureCode.MODEL_TIMEOUT, + at = STARTED_AT, + invocation = ExtractionInvocationId(invocationIndex = 1, attempt = 2), + ), + ), + ) + }.withMessageContaining("no invocation record for") + + // A failure outside any model call names none, and a run with no invocations still takes it. + assertThat( + runWith( + failures = listOf( + ExtractionFailure(ExtractionFailureCode.SOURCE_UNAVAILABLE, "source read failed", STARTED_AT), + ), + ).failures.single().invocation, + ).isNull() + } + + @Test + fun `an empty mutable collection handed in is still copied`() { + // The bug this is here for: a defensive copy skipped when the collection is empty leaves + // the run aliasing the caller's list, and the caller fills it afterwards. + val revisions = mutableListOf() + val invocations = mutableListOf() + val failures = mutableListOf() + + val run = ExtractionRun( + contextId = CONTEXT, + lineage = ExtractionRunLineage.root(RUN), + status = ExtractionRunStatus.RUNNING, + startedAt = STARTED_AT, + sourceRevisions = revisions, + invocations = invocations, + failures = failures, + ) + + revisions += REVISION_ONE + invocations += ExtractionInvocationRecord.planned(0) + failures += ExtractionFailure(ExtractionFailureCode.INTERNAL, "later", STARTED_AT) + + assertThat(run.sourceRevisions).isEmpty() + assertThat(run.invocations).isEmpty() + assertThat(run.failures).isEmpty() + } + + @Test + fun `a populated mutable collection handed in is copied too`() { + val revisions = mutableListOf(REVISION_ONE) + + val run = runWith(sourceRevisions = revisions) + revisions += REVISION_TWO + + assertThat(run.sourceRevisions).containsExactly(REVISION_ONE) + } + + @Test + @Suppress("UNCHECKED_CAST") + fun `the collections a run hands back cannot be edited`() { + val run = ExtractionRunFixtures.populatedRun() + + assertThrows { + (run.sourceRevisions as MutableList).add(REVISION_ONE) + } + assertThrows { + (run.invocations as MutableList).clear() + } + assertThrows { + (run.failures as MutableList).clear() + } + } + + @Test + fun `two runs built from the same values are equal and hash alike`() { + val one = ExtractionRunFixtures.populatedRun() + val two = ExtractionRunFixtures.populatedRun() + + assertThat(one).isEqualTo(two) + assertThat(one.hashCode()).isEqualTo(two.hashCode()) + assertThat(one).isEqualTo(one) + assertThat(one).isNotEqualTo(null) + assertThat(one).isNotEqualTo("not a run") + assertThat(setOf(one, two)).hasSize(1) + } + + @Test + fun `changing any single component makes it a different run`() { + val base = ExtractionRunFixtures.populatedRun() + + // Hand-written equality has to cover every component, so every component gets varied. A + // field left out of equals shows up here as an equal pair that should not be. + val variants: Map = mapOf( + "contextId" to base.copyWith(contextId = OTHER_CONTEXT), + "lineage" to base.copyWith(lineage = ExtractionRunLineage.root(ExtractionRunRef("run-other"))), + "status" to base.copyWith(status = ExtractionRunStatus.CANCELLED), + "startedAt" to base.copyWith(startedAt = STARTED_AT.minusSeconds(60)), + "finishedAt" to base.copyWith(finishedAt = null), + "profile" to base.copyWith(profile = ExtractionContentProfileRef("house-style", "v4")), + "sourceRevisions" to base.copyWith(sourceRevisions = listOf(REVISION_ONE)), + "fingerprints" to base.copyWith(fingerprints = ExtractionRunFingerprints()), + "runtime" to base.copyWith(runtime = ExtractionRuntimeIdentity()), + "requestedModel" to base.copyWith(requestedModel = ExtractionRequestedModelConfig()), + "subjectRefs" to base.copyWith(subjectRefs = ExtractionRunSubjectRefs()), + "experimentRef" to base.copyWith(experimentRef = null), + "cohortRef" to base.copyWith(cohortRef = null), + "replayFidelity" to base.copyWith(replayFidelity = ExtractionReplayFidelity.METADATA), + "counts" to base.copyWith(counts = ExtractionRunCounts()), + // Drops the first record and keeps the one the failure names, since a run rejects a + // failure whose invocation it has no record of. + "invocations" to base.copyWith(invocations = base.invocations.drop(1)), + "failures" to base.copyWith(failures = emptyList()), + ) + + assertThat(variants).hasSize(17) + variants.forEach { (component, variant) -> + assertThat(variant) + .describedAs("a run differing only in %s", component) + .isNotEqualTo(base) + } + } + + @Test + fun `the same run id in two tenants is two runs`() { + val here = ExtractionRunFixtures.startedRun(contextId = CONTEXT) + val there = ExtractionRunFixtures.startedRun(contextId = OTHER_CONTEXT) + + assertThat(here.key()).isNotEqualTo(there.key()) + assertThat(here).isNotEqualTo(there) + assertThat(here.key()).isEqualTo(ExtractionRunKey(CONTEXT, RUN)) + assertThat(here.key().getContextIdValue()).isEqualTo("tenant-4f2c9a") + assertThat(here.getContextIdValue()).isEqualTo("tenant-4f2c9a") + assertThat(setOf(here.key(), there.key())).hasSize(2) + } + + @Test + fun `the Java-facing factory takes the tenant as a string and matches the constructor`() { + val fromFactory = ExtractionRun.of( + contextIdValue = "tenant-4f2c9a", + lineage = ExtractionRunLineage.root(RUN), + status = ExtractionRunStatus.RUNNING, + startedAt = STARTED_AT, + ) + + assertThat(fromFactory).isEqualTo(ExtractionRunFixtures.startedRun()) + assertThat(fromFactory.contextId).isEqualTo(ContextId("tenant-4f2c9a")) + } + + private fun runWith( + sourceRevisions: List = emptyList(), + invocations: List = emptyList(), + failures: List = emptyList(), + ): ExtractionRun = ExtractionRun( + contextId = CONTEXT, + lineage = ExtractionRunLineage.root(RUN), + status = ExtractionRunStatus.RUNNING, + startedAt = STARTED_AT, + sourceRevisions = sourceRevisions, + invocations = invocations, + failures = failures, + ) +} + +/** + * Rebuilds a run with one component replaced. + * + * [ExtractionRun] is a plain class, so it has no generated `copy`; the equality test needs one and + * this keeps it in the test rather than widening the API for it. + */ +@Suppress("LongParameterList") +private fun ExtractionRun.copyWith( + contextId: ContextId = this.contextId, + lineage: ExtractionRunLineage = this.lineage, + status: ExtractionRunStatus = this.status, + startedAt: java.time.Instant = this.startedAt, + finishedAt: java.time.Instant? = this.finishedAt, + profile: ExtractionContentProfileRef? = this.profile, + sourceRevisions: List = this.sourceRevisions, + fingerprints: ExtractionRunFingerprints = this.fingerprints, + runtime: ExtractionRuntimeIdentity = this.runtime, + requestedModel: ExtractionRequestedModelConfig? = this.requestedModel, + subjectRefs: ExtractionRunSubjectRefs = this.subjectRefs, + experimentRef: ExtractionExperimentRef? = this.experimentRef, + cohortRef: ExtractionCohortRef? = this.cohortRef, + replayFidelity: ExtractionReplayFidelity = this.replayFidelity, + counts: ExtractionRunCounts = this.counts, + invocations: List = this.invocations, + failures: List = this.failures, +): ExtractionRun = ExtractionRun( + contextId = contextId, + lineage = lineage, + status = status, + startedAt = startedAt, + finishedAt = finishedAt, + profile = profile, + sourceRevisions = sourceRevisions, + fingerprints = fingerprints, + runtime = runtime, + requestedModel = requestedModel, + subjectRefs = subjectRefs, + experimentRef = experimentRef, + cohortRef = cohortRef, + replayFidelity = replayFidelity, + counts = counts, + invocations = invocations, + failures = failures, +) diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunFixtures.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunFixtures.kt new file mode 100644 index 00000000..3d0757d4 --- /dev/null +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunFixtures.kt @@ -0,0 +1,185 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import com.embabel.agent.core.ContextId +import com.embabel.dice.provenance.SourceRevisionRef +import java.time.Duration +import java.time.Instant + +/** + * Shared material for the extraction-run tests. + * + * The instants are fixed rather than `now()` so a dump of a populated run is byte-stable and so no + * nanosecond field turns into a nine-digit run that the privacy assertions would flag. + */ +internal object ExtractionRunFixtures { + + /** + * The source text the sanitization tests extract from. + * + * It carries a person's name, an organisation, an email address and a case number on purpose: + * those are the shapes that survive into a provider's exception message and then into a stored + * failure record if anything writes `e.message` into one. + */ + const val SOURCE_TEXT: String = + "Marguerite Okonkwo confirmed the Q3 renewal for Acme Holdings on 12 March 2026; " + + "reach her at marguerite.okonkwo@acme-holdings.example about case AB-7741-XZ." + + /** Distinctive pieces of [SOURCE_TEXT], each of which must not appear in a stored run. */ + val SOURCE_TEXT_FRAGMENTS: List = listOf( + "Marguerite", + "Okonkwo", + "Acme Holdings", + "Q3 renewal", + "AB-7741-XZ", + "marguerite.okonkwo@acme-holdings.example", + "12 March 2026", + ) + + val CONTEXT: ContextId = ContextId("tenant-4f2c9a") + val OTHER_CONTEXT: ContextId = ContextId("tenant-9b31de") + + val RUN: ExtractionRunRef = ExtractionRunRef("run-01JAV7Q2N4") + val PARENT_RUN: ExtractionRunRef = ExtractionRunRef("run-01JAV6M0K1") + val SUPERSEDED_RUN: ExtractionRunRef = ExtractionRunRef("run-01JAV5C8H7") + + val STARTED_AT: Instant = Instant.parse("2026-08-31T10:15:30Z") + val FINISHED_AT: Instant = Instant.parse("2026-08-31T10:15:47Z") + + val REVISION_ONE: SourceRevisionRef = SourceRevisionRef("uri:doc-a", "rev-11") + val REVISION_TWO: SourceRevisionRef = SourceRevisionRef("uri:doc-b", "rev-4") + + /** A minimal run: started, nothing else known. */ + fun startedRun( + contextId: ContextId = CONTEXT, + runRef: ExtractionRunRef = RUN, + ): ExtractionRun = ExtractionRun( + contextId = contextId, + lineage = ExtractionRunLineage.root(runRef), + status = ExtractionRunStatus.RUNNING, + startedAt = STARTED_AT, + ) + + /** + * A run with every field populated, which is what the privacy assertions dump. + * + * The failure on it is built the way DICE builds one — from a throwable whose message quotes + * [SOURCE_TEXT], exactly as a provider's would. + */ + fun populatedRun(): ExtractionRun = ExtractionRun( + contextId = CONTEXT, + lineage = ExtractionRunLineage.childOf( + runRef = RUN, + parent = ExtractionRunLineage.root(PARENT_RUN), + supersedesRunRef = SUPERSEDED_RUN, + ), + status = ExtractionRunStatus.FAILED, + startedAt = STARTED_AT, + finishedAt = FINISHED_AT, + profile = ExtractionContentProfileRef("house-style", "v3"), + sourceRevisions = listOf(REVISION_ONE, REVISION_TWO), + fingerprints = ExtractionRunFingerprints( + promptTemplateFingerprint = "sha256:6d1f0a2b", + schemaFingerprint = "sha256:a7c40e19", + metamodelFingerprint = "sha256:0b93cc55", + ), + runtime = ExtractionRuntimeIdentity( + extractor = "LlmPropositionExtractor", + extractorVersion = "0.2.0", + hostApplication = "assistant", + runtime = "dice", + runtimeVersion = "0.2.0", + ), + requestedModel = ExtractionRequestedModelConfig( + modelRole = "extraction", + requestedModel = "model-large", + temperature = 0.2, + topP = 0.9, + topK = 40, + maxTokens = 2048, + presencePenalty = 0.0, + frequencyPenalty = 0.1, + thinkingFingerprint = "think:8fa2", + selectionFingerprint = "select:11de", + timeout = Duration.ofSeconds(30), + ), + subjectRefs = ExtractionRunSubjectRefs( + actor = ExtractionActorRef("actor:7f19aa02"), + request = ExtractionRequestRef("req:5c8e1d34"), + session = ExtractionSessionRef("sess:2b70ffa9"), + personalization = ExtractionPersonalizationRef("pers:e14c7b60"), + deployment = ExtractionDeploymentRef("deploy:eu-west-1.blue"), + ), + experimentRef = ExtractionExperimentRef("exp:prompt-v3"), + cohortRef = ExtractionCohortRef("cohort:treatment"), + replayFidelity = ExtractionReplayFidelity.strongest(), + counts = ExtractionRunCounts( + sourcesRead = 2, + chunksProcessed = 6, + propositionsExtracted = 14, + propositionsPersisted = 11, + propositionsRejected = 3, + entitiesResolved = 9, + ), + invocations = listOf( + ExtractionInvocationRecord( + id = ExtractionInvocationId.planned(0), + outcome = ExtractionInvocationOutcome.SUCCEEDED, + configuredService = "service-alpha", + startedAt = STARTED_AT, + finishedAt = STARTED_AT.plusSeconds(4), + usage = ExtractionModelUsage( + inputTokens = 1820, + outputTokens = 344, + totalTokens = 2164, + cachedInputTokens = 512, + reasoningTokens = 96, + ), + providerResponse = ExtractionProviderResponseFacts( + responseModel = "model-large-2026-07", + responseId = "resp:c40d19ab", + finishReason = "stop", + systemFingerprint = "fp:31ac70", + ), + ), + ExtractionInvocationRecord( + id = ExtractionInvocationId(invocationIndex = 1, attempt = 2), + outcome = ExtractionInvocationOutcome.FAILED, + configuredService = "service-beta", + startedAt = STARTED_AT.plusSeconds(5), + finishedAt = FINISHED_AT, + ), + ), + failures = listOf( + ExtractionFailure.fromThrowable( + code = ExtractionFailureCode.DECODE_FAILED, + throwable = providerFailureQuotingSource(), + at = FINISHED_AT, + invocation = ExtractionInvocationId(invocationIndex = 1, attempt = 2), + ), + ), + ) + + /** + * The exception a provider throws when a call over [SOURCE_TEXT] goes wrong: the prompt, and + * therefore the source text, is quoted back in the message, and again in the cause. + */ + fun providerFailureQuotingSource(): Throwable = IllegalStateException( + "decode failed for prompt: $SOURCE_TEXT", + IllegalArgumentException("unexpected token near '$SOURCE_TEXT'"), + ) +} diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLineageTest.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLineageTest.kt new file mode 100644 index 00000000..f1e7ede8 --- /dev/null +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLineageTest.kt @@ -0,0 +1,155 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import org.assertj.core.api.Assertions.assertThat +import org.assertj.core.api.Assertions.assertThatIllegalArgumentException +import org.junit.jupiter.api.Test + +/** + * The root reference is denormalized, so it is only worth having if it cannot drift: a parentless + * run is its own root, a child takes its parent's root, and the constructor rejects every + * combination that would make the field say otherwise. + */ +class ExtractionRunLineageTest { + + private val runA = ExtractionRunRef("run-a") + private val runB = ExtractionRunRef("run-b") + private val runC = ExtractionRunRef("run-c") + private val runD = ExtractionRunRef("run-d") + + @Test + fun `a parentless run is its own root`() { + val lineage = ExtractionRunLineage.root(runA) + + assertThat(lineage.rootRunRef).isEqualTo(runA) + assertThat(lineage.parentRunRef).isNull() + assertThat(lineage.isRoot).isTrue() + assertThat(lineage.passIndex).isZero() + } + + @Test + fun `a child takes its parent's root and the next pass index`() { + val parent = ExtractionRunLineage.root(runA) + + val child = ExtractionRunLineage.childOf(runB, parent) + + assertThat(child.parentRunRef).isEqualTo(runA) + assertThat(child.rootRunRef).isEqualTo(runA) + assertThat(child.isRoot).isFalse() + assertThat(child.passIndex).isEqualTo(1) + } + + @Test + fun `the root carries down a chain rather than being recomputed at each hop`() { + val first = ExtractionRunLineage.root(runA) + val second = ExtractionRunLineage.childOf(runB, first) + val third = ExtractionRunLineage.childOf(runC, second) + val fourth = ExtractionRunLineage.childOf(runD, third) + + // Every run in the lineage names the same root, so "everything in this lineage" is one + // indexed read instead of a walk back up the parent chain. + assertThat(listOf(first, second, third, fourth).map { it.rootRunRef }) + .containsExactly(runA, runA, runA, runA) + assertThat(listOf(first, second, third, fourth).map { it.parentRunRef }) + .containsExactly(null, runA, runB, runC) + assertThat(listOf(first, second, third, fourth).map { it.passIndex }) + .containsExactly(0, 1, 2, 3) + } + + @Test + fun `parent and supersession are separate axes`() { + val parent = ExtractionRunLineage.root(runA) + + val replacementOfASibling = ExtractionRunLineage.childOf( + runRef = runB, + parent = parent, + supersedesRunRef = runC, + ) + val replacementWithNoParent = ExtractionRunLineage.root( + runRef = runD, + supersedesRunRef = runC, + ) + + assertThat(replacementOfASibling.parentRunRef).isEqualTo(runA) + assertThat(replacementOfASibling.supersedesRunRef).isEqualTo(runC) + // A re-extraction replaces an earlier run without continuing it: superseded, no parent, + // and still its own root. + assertThat(replacementWithNoParent.parentRunRef).isNull() + assertThat(replacementWithNoParent.supersedesRunRef).isEqualTo(runC) + assertThat(replacementWithNoParent.rootRunRef).isEqualTo(runD) + } + + @Test + fun `a parentless run whose root is some other run is rejected`() { + assertThatIllegalArgumentException().isThrownBy { + ExtractionRunLineage(runRef = runA, rootRunRef = runB) + }.withMessageContaining("no parent is its own root") + } + + @Test + fun `a run with a parent cannot claim to be its own root`() { + assertThatIllegalArgumentException().isThrownBy { + ExtractionRunLineage(runRef = runA, rootRunRef = runA, parentRunRef = runB) + }.withMessageContaining("takes its parent's root") + } + + @Test + fun `a run cannot be its own parent or supersede itself`() { + assertThatIllegalArgumentException().isThrownBy { + ExtractionRunLineage(runRef = runA, rootRunRef = runB, parentRunRef = runA) + }.withMessageContaining("its own parent") + + assertThatIllegalArgumentException().isThrownBy { + ExtractionRunLineage(runRef = runA, rootRunRef = runA, supersedesRunRef = runA) + }.withMessageContaining("supersede itself") + } + + @Test + fun `a negative pass index is rejected`() { + assertThatIllegalArgumentException().isThrownBy { + ExtractionRunLineage.root(runA, passIndex = -1) + }.withMessageContaining("passIndex") + } + + @Test + fun `copying a lineage still has to satisfy the root invariant`() { + val child = ExtractionRunLineage.childOf(runB, ExtractionRunLineage.root(runA)) + + // A data class copy runs the same init block, so nothing can drop the parent and keep a + // root that no longer follows from it. + assertThatIllegalArgumentException().isThrownBy { + child.copy(parentRunRef = null) + }.withMessageContaining("no parent is its own root") + } + + @Test + fun `the run exposes its lineage without a walk`() { + val run = ExtractionRun( + contextId = ExtractionRunFixtures.CONTEXT, + lineage = ExtractionRunLineage.childOf(runB, ExtractionRunLineage.root(runA)), + status = ExtractionRunStatus.RUNNING, + startedAt = ExtractionRunFixtures.STARTED_AT, + ) + + assertThat(run.ref).isEqualTo(runB) + assertThat(run.rootRef).isEqualTo(runA) + assertThat(run.parentRef).isEqualTo(runA) + assertThat(run.isRoot).isFalse() + assertThat(ExtractionRunFixtures.startedRun().isRoot).isTrue() + assertThat(ExtractionRunFixtures.startedRun().rootRef).isEqualTo(ExtractionRunFixtures.RUN) + } +} diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunPrivacyTest.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunPrivacyTest.kt new file mode 100644 index 00000000..74ab7b18 --- /dev/null +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunPrivacyTest.kt @@ -0,0 +1,279 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import com.embabel.dice.proposition.extraction.ExtractionRunFixtures.SOURCE_TEXT +import com.embabel.dice.proposition.extraction.ExtractionRunFixtures.SOURCE_TEXT_FRAGMENTS +import org.assertj.core.api.Assertions.assertThat +import org.assertj.core.api.Assertions.assertThatIllegalArgumentException +import org.junit.jupiter.api.Test +import java.lang.reflect.Field +import java.lang.reflect.Modifier + +/** + * What a stored run may and may not contain. + * + * The strong assertions here run over a *serialized row*: every field of a fully populated run, + * reached by reflection rather than through `toString`, so a field the summary leaves out is still + * covered. `toString` gets its own, weaker check, because a run reaches logs that way. + * + * The honest limit of these tests: a value type can bound a string and restrict its characters, and + * it can refuse to read an exception message. It cannot tell a pseudonym from a username. So the + * assertions are about what the types enforce — shapes that cannot be stored, and the one path + * DICE itself uses being incapable of carrying source text — not about a claim that no host can + * ever put something regrettable in a token. + */ +class ExtractionRunPrivacyTest { + + private val emailShape = Regex("[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}") + private val urlShape = Regex("(?i)\\b(?:https?|ftp|file|jdbc|bolt|neo4j)://") + private val longDigitRun = Regex("\\d{9,}") + + @Test + fun `a failure built from a provider exception carries none of the source text`() { + // The leak this guards: extraction runs over known material, the provider throws, and its + // message quotes the prompt — which is the source text — back at us. + val thrown = runCatching { extractPropositionsFrom(SOURCE_TEXT) }.exceptionOrNull()!! + assertThat(thrown.message).contains("Marguerite Okonkwo") + assertThat(thrown.cause?.message).contains("AB-7741-XZ") + + val failure = ExtractionFailure.fromThrowable(ExtractionFailureCode.DECODE_FAILED, thrown) + + assertThat(failure.detail) + .isEqualTo("java.lang.IllegalStateException <- java.lang.IllegalArgumentException") + SOURCE_TEXT_FRAGMENTS.forEach { fragment -> + assertThat(failure.detail).doesNotContain(fragment) + } + } + + @Test + fun `no fragment of the source text survives into a stored run`() { + val row = serializedRow(ExtractionRunFixtures.populatedRun()) + + // The run's only failure was built from the exception above, so if any part of the message + // path leaked, one of these fragments would be in the row. + SOURCE_TEXT_FRAGMENTS.forEach { fragment -> + assertThat(row).doesNotContain(fragment) + } + assertThat(row).doesNotContain(SOURCE_TEXT) + // The row really did reach the failure record, so the assertion above is not vacuous. + assertThat(row).contains("DECODE_FAILED", "java.lang.IllegalStateException") + } + + @Test + fun `a populated run holds no address, no link, and no long identifier run`() { + val row = serializedRow(ExtractionRunFixtures.populatedRun()) + + assertThat(emailShape.find(row)?.value).isNull() + assertThat(urlShape.find(row)?.value).isNull() + assertThat(longDigitRun.find(row)?.value).isNull() + // The row is a real dump: it reaches the tokens, the fingerprints and the counts. + assertThat(row).contains("actor:7f19aa02", "sha256:6d1f0a2b", "propositionsPersisted=11") + } + + @Test + fun `toString shows identity and sizes and none of the payload`() { + val rendered = ExtractionRunFixtures.populatedRun().toString() + + assertThat(rendered).contains("runId=run-01JAV7Q2N4", "rootRunId=run-01JAV6M0K1", "status=FAILED") + assertThat(rendered).contains("sourceRevisions=2", "invocations=2", "failures=1") + // Not in the summary: the tokens, the digests, the failure detail. + assertThat(rendered).doesNotContain("actor:7f19aa02", "sha256:6d1f0a2b", "IllegalStateException") + SOURCE_TEXT_FRAGMENTS.forEach { fragment -> assertThat(rendered).doesNotContain(fragment) } + } + + @Test + fun `an opaque token cannot be an address, a link, a path, or a name`() { + val rejected = listOf( + "marguerite.okonkwo@acme-holdings.example", + "https://acme-holdings.example/users/4471", + "/var/run/secrets/token", + "C:\\Users\\marguerite", + "Marguerite Okonkwo", + """{"userId":"4471"}""", + "token with spaces", + "line\nbreak", + ) + + rejected.forEach { candidate -> + assertThatIllegalArgumentException() + .describedAs("token %s", candidate) + .isThrownBy { ExtractionActorRef(candidate) } + } + + // What a host should mint instead: uuids, ULIDs, hex digests, namespaced opaque ids. + listOf( + "9f2a4c1e-3b77-4d0a-9c11-0a4e2b6d8f31", + "01JAV7Q2N4KX8ZP3W6Y5M0R7TB", + "actor:7f19aa02", + "sha256_6d1f0a2b3c4d", + "~tilde.and-dash_ok", + ).forEach { candidate -> assertThat(ExtractionActorRef(candidate).token).isEqualTo(candidate) } + } + + @Test + fun `a rejected token never appears in the message that rejects it`() { + // A validation failure propagates into logs, and the value that failed validation is + // exactly the one nobody vouched for. + val secretish = "marguerite.okonkwo@acme-holdings.example" + val overLong = "a".repeat(ExtractionRunLimits.MAX_IDENTIFIER_LENGTH + 1) + + val badShape = runCatching { ExtractionSessionRef(secretish) }.exceptionOrNull()!! + val tooLong = runCatching { ExtractionSessionRef(overLong) }.exceptionOrNull()!! + + assertThat(badShape.message).doesNotContain(secretish) + assertThat(badShape.message).contains("ExtractionSessionRef") + assertThat(tooLong.message).doesNotContain(overLong) + assertThat(tooLong.message).contains("257") + } + + @Test + fun `a token is bounded and blank is not a token`() { + val atCap = "a".repeat(ExtractionRunLimits.MAX_IDENTIFIER_LENGTH) + + assertThat(ExtractionDeploymentRef(atCap).token).hasSize(ExtractionRunLimits.MAX_IDENTIFIER_LENGTH) + assertThatIllegalArgumentException().isThrownBy { ExtractionDeploymentRef(atCap + "a") } + assertThatIllegalArgumentException().isThrownBy { ExtractionDeploymentRef(" ") } + } + + @Test + fun `toString on a token shows a prefix`() { + val ref = ExtractionPersonalizationRef("pers:e14c7b60cafe") + + assertThat(ref.toString()).isEqualTo("ExtractionPersonalizationRef(token=pers:e14…)") + assertThat(ref.toString()).doesNotContain("cafe") + } + + @Test + fun `two kinds of reference holding the same token are different references`() { + val actor = ExtractionActorRef("shared-token") + val session = ExtractionSessionRef("shared-token") + + assertThat(actor).isNotEqualTo(session) + assertThat(session).isNotEqualTo(actor) + assertThat(actor).isEqualTo(ExtractionActorRef("shared-token")) + assertThat(actor.hashCode()).isEqualTo(ExtractionActorRef("shared-token").hashCode()) + assertThat(setOf(actor, session)).hasSize(2) + } + + @Test + fun `a caller-written failure detail is flattened and clipped, and DICE says it cannot vouch for it`() { + val wordy = "chunk 3 of 12\n exceeded the budget\t" + "x".repeat(1_000) + + val failure = ExtractionFailure.of(ExtractionFailureCode.SCHEMA_VIOLATION, wordy) + + assertThat(failure.detail).hasSize(ExtractionRunLimits.MAX_FAILURE_DETAIL_LENGTH) + assertThat(failure.detail).startsWith("chunk 3 of 12 exceeded the budget x") + assertThat(failure.detail).doesNotContain("\n") + + // The constructor is stricter than the factory: it rejects rather than clipping, so a + // failure record can never be constructed over the bound by accident. + assertThatIllegalArgumentException().isThrownBy { + ExtractionFailure(ExtractionFailureCode.SCHEMA_VIOLATION, "y".repeat(513)) + } + assertThatIllegalArgumentException().isThrownBy { + ExtractionFailure(ExtractionFailureCode.SCHEMA_VIOLATION, "two\nlines") + }.withMessageContaining("single line") + } + + @Test + fun `a cause chain is bounded and a self-referencing cause terminates`() { + val deep = (1..12).fold(RuntimeException("root") as Throwable) { cause, _ -> + IllegalStateException("wrapper", cause) + } + val failure = ExtractionFailure.fromThrowable(ExtractionFailureCode.INTERNAL, deep) + + assertThat(failure.detail.split(" <- ")).hasSize(ExtractionFailure.MAX_CAUSE_CHAIN) + + val selfCausing = SelfCausingException() + assertThat( + ExtractionFailure.fromThrowable(ExtractionFailureCode.INTERNAL, selfCausing).detail, + ).isEqualTo(SelfCausingException::class.java.name) + } + + @Test + fun `no replay fidelity value promises exact replay`() { + val claimsTooMuch = listOf( + "EXACT", "DETERMINISTIC", "REPRODUCIBLE", "IDENTICAL", "GUARANTEED", "FULL", "COMPLETE", "PERFECT", + ) + + ExtractionReplayFidelity.entries.forEach { fidelity -> + claimsTooMuch.forEach { word -> + assertThat(fidelity.name).doesNotContain(word) + } + } + assertThat(ExtractionReplayFidelity.entries.map { it.name }) + .containsExactly("NONE", "METADATA", "APPROXIMATE") + // The strongest value the model offers is named for what it is. + assertThat(ExtractionReplayFidelity.strongest()).isEqualTo(ExtractionReplayFidelity.APPROXIMATE) + } + + /** + * Stands in for the extraction that would have produced this run: it reads known source text + * and fails the way a provider fails, with the prompt quoted back. + * + * There is no run coordinator until a later slice, so nothing here can drive a real extraction + * to a stored run. What this reproduces is the specific path a leak takes. + */ + private fun extractPropositionsFrom(sourceText: String): Nothing = + throw IllegalStateException( + "decode failed for prompt: $sourceText", + IllegalArgumentException("unexpected token near '$sourceText'"), + ) + + private class SelfCausingException : RuntimeException("self") { + override val cause: Throwable get() = this + } + + /** + * Renders every field of a value, recursively, the way a row writer would see it. + * + * Deliberately not `toString`: the point is to see fields a summary omits. It recurses into + * DICE types by reflection and renders anything else with `toString`, since JDK internals are + * neither ours nor reachable. + */ + private fun serializedRow(value: Any?, depth: Int = 0): String { + if (value == null) return "null" + if (depth > 16) return "…" + return when { + value is Enum<*> -> value.name + value is CharSequence || value is Number || value is Boolean || value is Char -> value.toString() + value is Iterable<*> -> value.joinToString(",", "[", "]") { serializedRow(it, depth + 1) } + value is Map<*, *> -> value.entries.joinToString(",", "{", "}") { + "${serializedRow(it.key, depth + 1)}=${serializedRow(it.value, depth + 1)}" + } + !value.javaClass.name.startsWith("com.embabel.") -> value.toString() + else -> instanceFields(value.javaClass).joinToString( + separator = ",", + prefix = "${value.javaClass.simpleName}{", + postfix = "}", + ) { field -> + field.isAccessible = true + "${field.name}=${serializedRow(field.get(value), depth + 1)}" + } + } + } + + private fun instanceFields(type: Class<*>): List { + val fields = mutableListOf() + var current: Class<*>? = type + while (current != null && current != Any::class.java) { + fields += current.declaredFields.filterNot { it.isSynthetic || Modifier.isStatic(it.modifiers) } + current = current.superclass + } + return fields + } +} diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunValueTypesTest.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunValueTypesTest.kt new file mode 100644 index 00000000..8b9ee367 --- /dev/null +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunValueTypesTest.kt @@ -0,0 +1,233 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import org.assertj.core.api.Assertions.assertThat +import org.assertj.core.api.Assertions.assertThatIllegalArgumentException +import org.junit.jupiter.api.Test +import java.time.Duration + +/** + * The construction and validation matrix for the smaller value types the run header is built from. + */ +class ExtractionRunValueTypesTest { + + private val overLong = "x".repeat(ExtractionRunLimits.MAX_IDENTIFIER_LENGTH + 1) + + @Test + fun `requested configuration accepts the portable fields`() { + val config = ExtractionRequestedModelConfig.of( + modelRole = "extraction", + requestedModel = "model-large", + temperature = 0.0, + topP = 1.0, + topK = 1, + maxTokens = 1, + presencePenalty = -2.0, + frequencyPenalty = 2.0, + thinkingFingerprint = "think:8fa2", + selectionFingerprint = "select:11de", + timeout = Duration.ofMillis(1), + ) + + assertThat(config.requestedModel).isEqualTo("model-large") + assertThat(config.temperature).isZero() + assertThat(config.timeout).isEqualTo(Duration.ofMillis(1)) + assertThat(ExtractionRequestedModelConfig()).isEqualTo(ExtractionRequestedModelConfig.of()) + } + + @Test + fun `requested configuration rejects values no provider would mean`() { + val rejections: List Any>> = listOf( + "blank role" to { ExtractionRequestedModelConfig(modelRole = " ") }, + "over-long model" to { ExtractionRequestedModelConfig(requestedModel = overLong) }, + "negative temperature" to { ExtractionRequestedModelConfig(temperature = -0.1) }, + "not-a-number temperature" to { ExtractionRequestedModelConfig(temperature = Double.NaN) }, + "topP above one" to { ExtractionRequestedModelConfig(topP = 1.1) }, + "topP below zero" to { ExtractionRequestedModelConfig(topP = -0.1) }, + "infinite penalty" to { + ExtractionRequestedModelConfig(presencePenalty = Double.POSITIVE_INFINITY) + }, + "zero topK" to { ExtractionRequestedModelConfig(topK = 0) }, + "zero maxTokens" to { ExtractionRequestedModelConfig(maxTokens = 0) }, + "zero timeout" to { ExtractionRequestedModelConfig(timeout = Duration.ZERO) }, + "negative timeout" to { ExtractionRequestedModelConfig(timeout = Duration.ofSeconds(-1)) }, + ) + + rejections.forEach { (name, construct) -> + assertThatIllegalArgumentException().describedAs(name).isThrownBy { construct() } + } + } + + @Test + fun `temperature has no upper bound because providers disagree on one`() { + // Some services stop at 1 and some at 2. Rejecting a legitimate 2.0 would be DICE deciding + // for a provider it never talks to. + assertThat(ExtractionRequestedModelConfig(temperature = 2.0).temperature).isEqualTo(2.0) + } + + @Test + fun `usage records what a provider reported and nothing derived`() { + val usage = ExtractionModelUsage.of(inputTokens = 100, outputTokens = 20, totalTokens = 7) + + // 100 + 20 is not 7, and that stands: an observed record records what was observed. + assertThat(usage.totalTokens).isEqualTo(7) + assertThat(ExtractionModelUsage().inputTokens).isNull() + + listOf<() -> Any>( + { ExtractionModelUsage(inputTokens = -1) }, + { ExtractionModelUsage(outputTokens = -1) }, + { ExtractionModelUsage(totalTokens = -1) }, + { ExtractionModelUsage(cachedInputTokens = -1) }, + { ExtractionModelUsage(reasoningTokens = -1) }, + ).forEach { construct -> assertThatIllegalArgumentException().isThrownBy { construct() } } + } + + @Test + fun `provider response facts stay absent rather than being filled in from the request`() { + val absent = ExtractionProviderResponseFacts() + + assertThat(absent.responseModel).isNull() + assertThat(absent.finishReason).isNull() + + val reported = ExtractionProviderResponseFacts.of( + responseModel = "model-large-2026-07", + responseId = "resp:c40d19ab", + finishReason = "stop", + systemFingerprint = "fp:31ac70", + ) + assertThat(reported.responseModel).isEqualTo("model-large-2026-07") + + assertThatIllegalArgumentException() + .isThrownBy { ExtractionProviderResponseFacts(responseId = overLong) } + assertThatIllegalArgumentException() + .isThrownBy { ExtractionProviderResponseFacts(finishReason = "") } + } + + @Test + fun `fingerprints and runtime identity are bounded identifiers`() { + assertThat( + ExtractionRunFingerprints.of(schemaFingerprint = "sha256:a7c40e19").schemaFingerprint, + ).isEqualTo("sha256:a7c40e19") + assertThat(ExtractionRunFingerprints()).isEqualTo(ExtractionRunFingerprints.of()) + assertThatIllegalArgumentException() + .isThrownBy { ExtractionRunFingerprints(promptTemplateFingerprint = overLong) } + assertThatIllegalArgumentException() + .isThrownBy { ExtractionRunFingerprints(metamodelFingerprint = " ") } + + assertThat(ExtractionRuntimeIdentity.of(extractor = "LlmPropositionExtractor").extractor) + .isEqualTo("LlmPropositionExtractor") + assertThatIllegalArgumentException() + .isThrownBy { ExtractionRuntimeIdentity(hostApplication = overLong) } + assertThatIllegalArgumentException() + .isThrownBy { ExtractionRuntimeIdentity(runtimeVersion = "") } + } + + @Test + fun `counts start at zero and never go below it`() { + assertThat(ExtractionRunCounts()).isEqualTo(ExtractionRunCounts.of()) + assertThat(ExtractionRunCounts().propositionsExtracted).isZero() + + listOf<() -> Any>( + { ExtractionRunCounts(sourcesRead = -1) }, + { ExtractionRunCounts(chunksProcessed = -1) }, + { ExtractionRunCounts(propositionsExtracted = -1) }, + { ExtractionRunCounts(propositionsPersisted = -1) }, + { ExtractionRunCounts(propositionsRejected = -1) }, + { ExtractionRunCounts(entitiesResolved = -1) }, + ).forEach { construct -> assertThatIllegalArgumentException().isThrownBy { construct() } } + } + + @Test + fun `a run may have some subject references and not others`() { + val partial = ExtractionRunSubjectRefs.of( + actor = ExtractionActorRef("actor:7f19aa02"), + deployment = ExtractionDeploymentRef("deploy:eu-west-1.blue"), + ) + + assertThat(partial.actor).isNotNull() + assertThat(partial.session).isNull() + assertThat(partial.personalization).isNull() + assertThat(ExtractionRunSubjectRefs()).isEqualTo(ExtractionRunSubjectRefs.of()) + } + + @Test + fun `a failure is a code, and everything else is optional`() { + val bare = ExtractionFailure(ExtractionFailureCode.MODEL_TIMEOUT) + + assertThat(bare.code).isEqualTo(ExtractionFailureCode.MODEL_TIMEOUT) + assertThat(bare.detail).isEmpty() + assertThat(bare.invocation).isNull() + + val tied = ExtractionFailure.of( + code = ExtractionFailureCode.RATE_LIMITED, + detail = "provider quota exhausted", + at = ExtractionRunFixtures.FINISHED_AT, + invocation = ExtractionInvocationId(2, 3), + ) + assertThat(tied.invocation).isEqualTo(ExtractionInvocationId(2, 3)) + assertThat(tied.detail).isEqualTo("provider quota exhausted") + } + + @Test + fun `every extraction run type is marked experimental`() { + // #67 will move these. The marker says so at the call site, not only in the CHANGELOG. + listOf( + ExtractionRun::class.java, + ExtractionRunKey::class.java, + ExtractionRunLineage::class.java, + ExtractionRunStatus::class.java, + ExtractionReplayFidelity::class.java, + ExtractionRunLimits::class.java, + ExtractionRunFingerprints::class.java, + ExtractionRuntimeIdentity::class.java, + ExtractionRunCounts::class.java, + ExtractionRunSubjectRefs::class.java, + ExtractionRequestedModelConfig::class.java, + ExtractionInvocationId::class.java, + ExtractionInvocationOutcome::class.java, + ExtractionInvocationRecord::class.java, + ExtractionModelUsage::class.java, + ExtractionProviderResponseFacts::class.java, + ExtractionFailure::class.java, + ExtractionFailureCode::class.java, + ExtractionOpaqueRef::class.java, + ExtractionActorRef::class.java, + ExtractionRequestRef::class.java, + ExtractionSessionRef::class.java, + ExtractionPersonalizationRef::class.java, + ExtractionDeploymentRef::class.java, + ExtractionExperimentRef::class.java, + ExtractionCohortRef::class.java, + ).forEach { type -> + assertThat(isMarkedExperimental(type)) + .describedAs("%s is marked experimental", type.simpleName) + .isTrue() + } + } + + /** + * `@ApiStatus.Experimental` has class retention, so reflection cannot see it at runtime. The + * marker is in the class file's constant pool either way, which is what this reads. + */ + private fun isMarkedExperimental(type: Class<*>): Boolean { + val bytes = checkNotNull(type.classLoader.getResourceAsStream(type.name.replace('.', '/') + ".class")) { + "no class file for ${type.name}" + }.use { it.readBytes() } + return String(bytes, Charsets.ISO_8859_1) + .contains("Lorg/jetbrains/annotations/ApiStatus\$Experimental;") + } +} diff --git a/docs/design/INDEX.md b/docs/design/INDEX.md index 32f99796..f2f8e724 100644 --- a/docs/design/INDEX.md +++ b/docs/design/INDEX.md @@ -25,6 +25,14 @@ you need. through extraction without DICE resolving it: why profile identity is opaque, why profile, perspective, schema and tenant stay independent, and why an extraction run reference shipped in review and was pulled back out until its consuming write exists. EXPERIMENTAL. + and a run reference through extraction without DICE resolving either: why profile identity is + opaque, why the run reference ships ahead of the run, and why profile, perspective, schema and + tenant stay independent. EXPERIMENTAL. +- [extraction-runs.md](extraction-runs.md) — the durable record of one extraction execution: why + requested model configuration and observed provider facts are separate types, how invocation + identity comes from the call plan rather than completion order, the denormalized root run + reference, the privacy contract on opaque references and sanitized failures, and why replay + fidelity is never exact. EXPERIMENTAL. ## Propositions & lifecycle diff --git a/docs/design/extraction-runs.md b/docs/design/extraction-runs.md new file mode 100644 index 00000000..c69114b9 --- /dev/null +++ b/docs/design/extraction-runs.md @@ -0,0 +1,320 @@ +# Extraction runs: what produced a claim, recorded without holding what it read + +An extraction run is the durable record of one execution: which profile, prompt, schema and +metamodel versions were in force, which revisions of which sources it read, what it asked a +model for, what the provider actually reported back, how far it got, and what went wrong. It +holds none of the material. No prompts, no source text, no responses, no user or session +objects, no provider SDK payloads, no extension maps. + +This note covers DICE #67's value model — the types in `com.embabel.dice.proposition.extraction` +that later slices store, key, and expose. The lifecycle state machine, the store contract, the +Drivine implementation, the proposition-to-run relation and the wiring are separate slices; where +this note says "the store contract", that is what it means. + +## What a run holds + +```mermaid +flowchart TD + RUN["ExtractionRun
keyed by (ContextId, ExtractionRunRef)"] + RUN --> LIN["ExtractionRunLineage
run, root, parent, supersedes, pass"] + RUN --> PROF["ExtractionContentProfileRef
#66"] + RUN --> SRC["List<SourceRevisionRef>
#64, ordered"] + RUN --> FP["ExtractionRunFingerprints
prompt, schema, metamodel"] + RUN --> RT["ExtractionRuntimeIdentity
extractor, host, runtime"] + RUN --> REQ["ExtractionRequestedModelConfig
what was asked for"] + RUN --> SUBJ["ExtractionRunSubjectRefs
actor, request, session,
personalization, deployment"] + RUN --> EXP["ExtractionExperimentRef
ExtractionCohortRef"] + RUN --> ST["ExtractionRunStatus
ExtractionReplayFidelity
ExtractionRunCounts"] + RUN --> INV["List<ExtractionInvocationRecord>
what was observed"] + RUN --> FAIL["List<ExtractionFailure>
bounded, sanitized"] + INV --> IID["ExtractionInvocationId
plan ordinal + attempt"] + INV --> USE["ExtractionModelUsage"] + INV --> PRF["ExtractionProviderResponseFacts"] + SUBJ --> OP["ExtractionOpaqueRef
bounded pseudonymous token"] + EXP --> OP +``` + +`ExtractionRunKey` is the tenant-qualified identity: a run id is host-minted and DICE never +assumes it is unique across tenants, so the `ContextId` travels with it. Two tenants that both +mint `run-1` have two runs. + +## Requested and observed are different types + +The single most useful thing a run can tell an incident is which of these two it is looking at: +the temperature the host asked for, or the temperature the provider used. A service can clamp a +temperature, ignore a top-k, cap max tokens below what was asked, and route to a different +checkpoint under a stable model name. None of that is visible to DICE, so a model that let one +field mean both would be inviting a false answer to the only question worth asking. + +So the split is structural, not conventional: + +| | Type | Lives on | Filled in by | +| --- | --- | --- | --- | +| Requested | `ExtractionRequestedModelConfig` | the run header | the host, before any call | +| Observed | `ExtractionInvocationRecord`, with `ExtractionModelUsage` and `ExtractionProviderResponseFacts` | one per attempt | whatever came back | + +An invocation record has no field of the requested type, and the two share no property name — the +requested one is `requestedModel`, the reported one is `responseModel`. Both are asserted by test +rather than left to review: a mapper that copied one into the other would have to be written on +purpose. + +The corollary is that an absent observed field stays absent. A run that asked for `model-large` +and got no model name back records a null `responseModel`, because telling "the provider did not +say" apart from "the provider said what we asked for" is the whole point of the field. + +`ExtractionRequestedModelConfig` carries portable fields only: model and role, temperature, top-p, +top-k, max tokens, the two penalties, a thinking fingerprint, a selection fingerprint, and a +timeout. There is no provider extension object, no settings blob, no free map. That is where +credentials, system prompts and whole SDK request bodies get persisted by accident. A +provider-specific knob a host cares about is folded into one of the fingerprints — an opaque +digest DICE compares and never reads. + +Ranges are checked where every provider agrees and left open where they do not. Temperature has no +upper bound because services differ on whether it stops at 1 or 2, and the penalties are only +required to be real numbers for the same reason. Rejecting a legitimate `2.0` would be DICE +deciding for a provider it never talks to. + +## Invocation identity comes from the plan, never from completion + +A run makes zero, one, or many model calls — chunking splits work, retries repeat it. Every +record needs an identity that a store can use as a deterministic child key, so that a retried +write lands on its own row and a replayed write upserts in place. + +The identity is allocated when the call plan is laid out, before the first request goes out: + +```mermaid +flowchart LR + P["plan(4)"] --> I0["id 0/1"] & I1["id 1/1"] & I2["id 2/1"] & I3["id 3/1"] + I2 -->|"returns first"| R2["record 2/1 SUCCEEDED"] + I0 -->|"returns second"| R0["record 0/1 SUCCEEDED"] + I1 -->|"fails"| F1["record 1/1 FAILED"] + F1 -->|"retry()"| RT["record 1/2"] +``` + +`invocationIndex` is the ordinal in the plan. `attempt` counts tries at that same call, from 1. +`retry()` carries the index forward, increments the attempt, and resets every observed field, +because those observations belonged to the attempt that just failed. + +Timing on a record is an observation like any other and may be absent even on a terminal outcome. A +`SUCCEEDED` attempt with no `startedAt` means the clock was not recorded, not that the call did not +run; requiring timing would push callers to invent a duration, which is worse evidence than none. +The two checks a record can fail on its own terms do apply: a finish cannot precede its start, and +an `IN_FLIGHT` attempt has not finished. + +Completion order writes into identities that already exist. There is no factory that takes a +position in a result list, and the run stores records in whatever order they arrived while +`invocationsInPlanOrder()` reads the plan back out. A run rejects two records with the same +`(invocationIndex, attempt)`. + +## The root run reference, and why it is denormalized + +`ExtractionRunLineage` carries four references: the run, its parent, what it supersedes, and its +root. Parent and supersession are separate axes — a parent is the run this one continues from, a +superseded run is one this one replaces — and a run can have one of each, both, or neither. + +The root is redundant with the parent chain, and it is stored anyway. OpenLineage's +`ParentRunFacet` does the same: it carries an optional `root` alongside the immediate parent so +consumers do not have to walk the chain a hop at a time. Deep pass-and-retry chains are exactly +where walking hurts, and the audit projection reads lineage by run. + +A denormalized field is only worth having if it cannot drift, so it is fixed at mint and the +constructor rejects every combination that would make it a lie: + +- a run with no parent is its own root; +- a run with a parent takes its parent's root, and therefore is not its own root; +- a run is neither its own parent nor its own supersession. + +`ExtractionRunLineage.root(...)` and `.childOf(...)` do the arithmetic, and `childOf` defaults the +pass index to the parent's plus one. + +What a value type cannot check is a cycle of length two or more: that needs the other runs, so +bounded cycle-safe traversal belongs to the store that walks the chains. + +## The privacy contract + +Five references say whose work a run was — actor, request, session, personalization, deployment — +and two more group runs for comparison: experiment and cohort. All seven are the same kind of +thing, `ExtractionOpaqueRef`: a bounded, host-minted token that DICE compares, stores, and never +parses. + +What a host takes on when it mints one: + +- it is a pseudonym, not an email address, a username, a phone number, a customer number or a name; +- it is not dereferenceable into anything sensitive — no URL, no signed link, no bearer token, no + API key, no cookie value; +- it carries no authorization, and DICE never presents it to anything; +- it is stable enough to group by and cheap enough to rotate. + +**What the type enforces, and what it cannot.** Construction bounds the length and restricts the +characters to `A-Z a-z 0-9 . _ : ~ -`. That excludes whitespace, control characters, `@`, `/` and +`\`, so an email address, a URL, a file path, a JSON fragment and a human name are all rejected +outright — the common shapes of a leaked identifier cannot be stored at all. It cannot tell a +pseudonym from a username: `jdunnam` and `55512345` both pass. The last mile is the host's, and +the KDoc says so in the same words rather than implying a guarantee the code does not make. + +Two smaller things fall out of the same reasoning. A token's `toString` shows the first eight +characters, so a reference does not spread through logs in full. And a validation message names the +field and the length and never quotes the value — an `IllegalArgumentException` propagates into +logs, and the value that failed validation is exactly the one nobody vouched for. + +### Failures + +A failure record is a classified code plus a short single-line detail, and the run holds at most 64 +of them. + +Failure records are where source text leaks. A provider quotes the prompt back in its exception +message; a decode error carries the fragment it choked on. Both land in a stored run header the +moment someone writes `e.message` into one. So `ExtractionFailure.fromThrowable` — the path DICE +itself uses — never reads `Throwable.message`. It records the exception class names down the cause +chain, bounded to five links and cycle-safe, and nothing else. That detail cannot contain source +text because it never touched any. + +`ExtractionFailure.of` exists for the case where a caller genuinely knows something useful +("chunk 3 of 12 exceeded the token budget"). DICE cannot check what a caller puts there; it bounds +it and flattens it to a single line so a pasted stack trace does not fit. + +The tests are built to match what is actually enforced. A fixture with known source text +(a person, an organisation, an email address, a case number) is fed through a provider-shaped +exception that quotes it; the resulting failure and a full field-by-field dump of the populated run +are asserted to contain none of its fragments, no address shape, no link shape, and no long digit +run. The dump is reflective rather than `toString`, so a field the summary omits is still covered — +and `run.toString()` gets its own check that it shows identity, state and sizes and none of the +tokens, digests or details. + +## Replay is approximate, and named that way + +`ExtractionReplayFidelity` has three values: `NONE`, `METADATA`, `APPROXIMATE`. The strongest one +is still approximate, and `strongest()` returns it so the honesty of the claim survives someone +appending a value later. + +There is no value meaning "run this again and get the same output". A hosted model can change +weights, quantization, routing, safety filtering and system instructions under a stable model +name, none of it visible to DICE. Temperature zero narrows the distribution and does not remove +batching and floating-point nondeterminism. The field says what the *record* supports — nothing +recorded, identities and fingerprints only, or those plus the requested configuration — and makes +no promise about the provider. Host replay policy stays the host's. + +## Four lifecycle states + +`ExtractionRunStatus` is `RUNNING`, `COMPLETED`, `FAILED`, `CANCELLED`. The values only: the +transitions, the compare-and-set rules and idempotent terminal rewrites belong to the store +contract, so this type does not half-encode them. It checks that a finish does not precede a start +and stops there — a terminal status with no finish time is constructible here and is the state +machine's to reject. + +`COMPLETED`'s meaning is pinned where the value is declared, because it is not obvious and it is +load-bearing: every product the run's request called for is either durably persisted or terminally +disposed. It is written after persistence, never before, so a run whose persistence never finished +stays `RUNNING` and retryable. A run with zero products terminalizes `COMPLETED` vacuously. + +MLflow's run status has five values — `RUNNING`, `SCHEDULED`, `FINISHED`, `FAILED`, `KILLED`. +The two DICE does not have are deliberate: + +- **`SCHEDULED`** has no writer. There is no scheduler in this design, so nothing can observe a run + between "requested" and "started"; the state would be defined and never used. +- **`KILLED`** folds into `CANCELLED`. The fact an operator or an audit acts on is that the run + stopped short of its products, and that is the same fact whichever side pressed stop. `CANCELLED` + is also the abandonment path for a partially successful run nobody intends to finish. + +## OpenTelemetry GenAI naming, not adopted + +OTel's GenAI semantic conventions cover the same ground — `gen_ai.request.*`, `gen_ai.response.*`, +`gen_ai.usage.*`, and an opt-in-only gate on content capture that matches this model's +"no payloads by default" stance almost exactly. The field names are still not adopted, for one +reason: as of mid-2026 every `gen_ai.*` attribute is at stability level Development, and in June +2026 the conventions were moved out of the main semantic-conventions repository into a dedicated +one. Pinning a stored schema to names that are still moving buys interop now and a migration later. + +The structural agreement is worth keeping in view. A future OTel-compatible export is a mapping +from these types onto whatever the conventions stabilise as, and this model has a field for each +of the ones that matter. That is a better position than having adopted a naming that then changed. + +## The cap rule + +Every string a run stores is bounded, the bound is a named constant on `ExtractionRunLimits`, the +check runs in the `init` block of the type that owns the value, and anything over the bound is +**rejected**. Truncating an identifier would be worse than rejecting it: a shortened id is a +different id, and a store would then key rows on a value the caller never minted. + +| Constant | Value | Applies to | +| --- | --- | --- | +| `MAX_IDENTIFIER_LENGTH` | 256 | opaque tokens, fingerprints, model and role names, service names, provider response ids, runtime identifiers | +| `MAX_SOURCE_KEY_LENGTH` | 1024 | source keys, which come out of `SourceLocator.key()` and can legitimately hold a long URL | +| `MAX_FAILURE_DETAIL_LENGTH` | 512 | the one free-text field | +| `MAX_SOURCE_REVISIONS` | 256 | source revisions per run | +| `MAX_INVOCATIONS` | 1024 | invocation records per run, across every call and attempt | +| `MAX_FAILURES` | 64 | failure records per run | + +The failure detail is the one exception to rejection, and only on the way in: the factories shorten +it before construction, because keeping a clipped failure record beats losing the failure. The +constructor still rejects a longer one. + +Lengths count UTF-16 chars, so a 256-char identifier can be around 1 KB of UTF-8. The bound exists +to keep a run header finite. + +`SourceRevisionRef` predates this rule and validates non-blank only, so `ExtractionRun` applies the +source-key and identifier bounds to the revisions it stores. That is a stopgap: the check belongs +on `SourceRevisionRef` itself, so the bound travels with the type instead of being re-applied by +every consumer. Moving it is a one-line change in the module that owns that type and is follow-up +work. + +One string sits outside the rule and stays outside it: `ContextId.value`, the tenant, which is +validated non-blank and not bounded. `ContextId` is a DICE-wide type owned by the agent framework, +so bounding it is not this model's call. It is worth naming because the tenant is half of +`ExtractionRunKey`, so the run store's key is bounded on one side only — whoever sizes that key's +index in the store slice decides what to do about the other side. + +Two bounds are enforced away from the field they protect, because the field is not where the cost +lands. `ExtractionInvocationRecord.plan(count)` checks `MAX_INVOCATIONS` against the count before +allocating anything: a plan size derived from chunking a large document can be enormous, and +learning that from the run's own bound would mean building the whole list first. And a run rejects +a failure whose `invocation` names an identity it holds no record of — a dangling reference reads +as evidence about a call and nothing can join it to one, so the pair arrives together or not at +all. A failure that happened outside any model call names no invocation and is always accepted. + +## Value-type discipline + +Everything here is immutable and validated in `init`, with `@JvmStatic`/`@JvmOverloads` factories +on the types that have optional parameters. + +`ExtractionRun` itself is a plain class rather than a data class, for two reasons. A data class has +to declare its collection parameters as properties, which means the field *is* the caller's list +and there is nowhere to copy it. And a generated `copy`/`componentN` surface would pin an ABI +across seventeen fields while #67 is still moving. Equality and hash are written out over every +component, and a test varies each of the seventeen in turn so a component dropped from `equals` +fails rather than passing quietly. + +Collections are copied on the way in **unconditionally**, empty ones included. A copy skipped when +the list is empty leaves the run aliasing a list the caller still holds, and the caller fills it +afterwards; it fails later and stranger than the non-empty case. The copies are unmodifiable, so +the list a caller reads back cannot be edited either. + +## Status: EXPERIMENTAL + +Every type in this slice carries `@ApiStatus.Experimental`, the marker DICE already uses for API +that may still move, and a test reads the class files to assert none was missed. The KDoc on each +type says the same in words and the CHANGELOG entry is labelled. + +As with #66, a Kotlin `@RequiresOptIn` marker would make the status enforceable at the call site +rather than advisory. DICE defines none today, and inventing one is a decision about the whole +public surface. + +## What this slice does not do + +- **No store.** Nothing persists a run. The store contract, the in-memory implementation and the + lifecycle state machine are the next slice; the Drivine implementation and its schema follow. +- **No lifecycle.** There are four status values and no transitions. Nothing here can move a run + from `RUNNING` to anything. +- **No coordinator.** Nothing constructs an `ExtractionRun` during extraction yet, so the + sanitization tests reproduce the leak path rather than driving a real extraction into a stored + run. +- **No proposition-to-run relation.** Attribution from a claim to the runs that produced or + confirmed it is its own slice, on canonical saved ids, and run identity stays out of + source-provenance equality. +- **No protected-content references.** Optional replay material represented by classified, + expiring references is part of #67 and is not in this slice; the model's current answer to + replay material is that there is none. +- **No per-invocation requested configuration.** The requested configuration is one record on the + run header. A later slice that needs to vary settings per call adds a separate requested record + keyed by invocation index rather than a field on the observed record, which would collapse the + distinction this slice exists to draw. From 61dab8c371cdbc306f8036e7077837d0f6af71ea Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Mon, 31 Aug 2026 16:59:02 -0400 Subject: [PATCH 2/7] Align run recording with the source revision contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run recording no longer caps sourceKey or sourceRevision length: the revision contract is opaque and non-blank, so a value a query accepts can never fail run recording. A real store constraint brings the bound back if one ever exists. ExtractionRunLineage's constructor goes private with copy() following it, leaving root() and childOf() as the only public mints — both derive the root from the parent's own record, so a root that contradicts the parent chain has no public way in. Reflection and a future deserializer still reach the constructor; the docs and a test state that residual until a store slice closes it. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com> --- CHANGELOG.md | 85 ++++---- .../ConversationAnalysisRequestEvent.kt | 17 +- .../dice/common/SourceAnalysisContext.kt | 20 +- .../dice/common/SourceAnalysisRequestEvent.kt | 13 +- .../extraction/ExtractionRequest.kt | 29 ++- .../proposition/extraction/ExtractionRun.kt | 16 +- .../extraction/ExtractionRunLimits.kt | 15 +- .../extraction/ExtractionRunLineage.kt | 63 +++++- .../IncrementalPropositionExtraction.kt | 13 +- .../dice/provenance/SourceRevisionRef.kt | 10 +- .../ExtractionProfileJavaInteropTest.java | 146 ++++++++++++-- .../ExtractionProfileCompatibilityTest.kt | 36 ++-- .../ExtractionContextIndependenceTest.kt | 29 +++ .../SourceAnalysisRequestEventProfileTest.kt | 20 +- .../ExtractionProfileContractTest.kt | 58 +++++- .../extraction/ExtractionRunContractTest.kt | 44 +++-- .../extraction/ExtractionRunLineageTest.kt | 97 +++++++--- .../IncrementalPropositionExtractionTest.kt | 182 ++++++++++-------- .../provenance/SourceRevisionContractTest.kt | 31 +++ docs/design/extraction-runs.md | 36 +++- 20 files changed, 697 insertions(+), 263 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9e763232..0e5d0aed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1167,32 +1167,34 @@ and the consumer PRs that deliver it). make that enforceable at the call site rather than advisory; DICE defines none today and the design note records it as an open question. - **A run reference travelled with this slice for one round and was pulled back out** (PR #94 - review). `ExtractionRunRef` shipped identity-only, ahead of the durable run store that would key - on it. Nothing on this branch consumes it: `persistAndProject`, the method that actually saves - extraction's output, takes only the pipeline's result and never sees the context that would have - carried a run reference, so a caller passing one got it silently accepted and then dropped. - `currentRun` reached the same `PropositionExtractor.extract` extension point `profile` does, so - the two were equally reachable; the difference was that a run id had no store to resolve against, - so a reader would have had nothing to act on even if one existed. Because that store does not - exist on this branch, `currentRun`/`ExtractionRunRef` are removed from this slice entirely — - `SourceAnalysisContext`, every `remember*` entry point, `SourceAnalysisRequestEvent`, and - `ConversationAnalysisRequestEvent` — and return together with the write that consumes them once - the durable run store lands (DICE #67 and the run-model slices above it). No caller outside this - slice's own code and tests used the parameter for anything, so there is nothing to migrate. When - the reference does return it arrives as a field on `ExtractionRequest`, which is what the request - object is for: the entry-point signatures will be the ones this entry describes. - slice's own code and tests used the parameter for anything, so there is nothing to migrate. - claimed for `SourceAnalysisContext`: two more fields rewrite `copy`, add two `componentN` - methods, and change the synthetic `$default` constructor, so Kotlin code compiled against an - earlier jar must be recompiled rather than swapped in — the same half of the boundary #64 - declined, pinned here by a test asserting exactly one `copy` remains and that it takes thirteen - arguments. No stored data changes and no migration is required: nothing serializes a profile or a - run reference yet. Extraction, resolution, and revision ordering are behaviour-identical; a - profile changes what a run is attributed to, not what it does. `ExtractionContentProfileRef` and - `ExtractionRunRef` both carry `@ApiStatus.Experimental` and their shapes may still move while - #67 lands. A Kotlin `@RequiresOptIn` marker would make that enforceable at the call site rather - than advisory; DICE defines none today and the design note records it as an open question. + **A run reference travelled with this slice for one round, was pulled back out, and has now + returned on the request** (PR #94 review, then PR #95). `ExtractionRunRef` first shipped + identity-only, as a loose parameter on every `remember*` entry point, ahead of the durable run + store that would key on it. Nothing consumed it: `persistAndProject`, the method that actually + saves extraction's output, takes only the pipeline's result and never sees the context that + would have carried a run reference, so a caller passing one got it silently accepted and then + dropped. The review's objection was to the parameters, so the parameters went, and + `ExtractionRequest` arrived to make the next dimension cost no signature at all. `currentRun` is + now a fourth field on that request, reaching `SourceAnalysisContext`, `SourceAnalysisRequestEvent` + and `ConversationAnalysisRequestEvent` the way a profile does — and the entry-point descriptor + sets are byte-for-byte the ones the paragraph above describes. That is the request object paying + for itself: a new extraction dimension landed with no new method, no new arity, and nothing for a + subclass or a Java caller to migrate. What has not changed is that nothing reads the run yet. It + is identity only until the durable run store lands (DICE #67 and the run-model slices above it); + a test pins that `persistAndProject` still has exactly one overload taking only the pipeline's + result, so there is still no write for a run reference to reach. + + Full Kotlin synthetic `copy` and `componentN` ABI is **not** claimed for `SourceAnalysisContext`: + two more fields rewrite `copy`, add two `componentN` methods, and change the synthetic `$default` + constructor, so Kotlin code compiled against an earlier jar must be recompiled rather than + swapped in — the same half of the boundary #64 declined, pinned here by a test asserting exactly + one `copy` remains and that it takes thirteen arguments. No stored data changes and no migration + is required: nothing serializes a profile or a run reference yet. Extraction, resolution, and + revision ordering are behaviour-identical; a profile changes what a run is attributed to, not + what it does. `ExtractionContentProfileRef` and `ExtractionRunRef` both carry + `@ApiStatus.Experimental` and their shapes may still move while #67 lands. A Kotlin + `@RequiresOptIn` marker would make that enforceable at the call site rather than advisory; DICE + defines none today and the design note records it as an open question. - **EXPERIMENTAL.** The extraction run model in `dice` core — the value types DICE #67's store, lineage and wiring slices build on. `ExtractionRun`, keyed by (`ContextId`, `ExtractionRunRef`) @@ -1226,10 +1228,17 @@ and the consumer PRs that deliver it). reference is denormalized.** `ExtractionRunLineage` carries the run, its parent, what it supersedes, its pass index, and its root, following OpenLineage's `ParentRunFacet`, which also ships a root alongside the immediate parent so consumers need not walk the chain. The root is - fixed at mint — a parentless run is its own root, a child takes its parent's root — and the - constructor rejects a parentless run whose root is another run, a child claiming to be its own - root, and self-parenting or self-supersession. Cycles longer than one need the other runs and stay - with the store that walks the chains. **Privacy is a contract with an enforced floor.** + fixed at mint — a parentless run is its own root, a child takes its parent's root. `root()` and + `childOf()` (`copy()` follows too, via `@ConsistentCopyVisibility`) are the only public way to + mint one: `root()` sets the root to the run's own ref, `childOf()` derives it from the actual + parent lineage it is handed, and either way no public parameter can carry a root that disagrees + with the parent chain. That closes the public API only. Kotlin reflection, and a Jackson + deserializer resolving the primary constructor the same way, can still construct one with a + contradictory root — `ExtractionRunLineageTest` pins the gap openly. Nothing serializes this type + today, so it affects a future store slice, not current wiring. Self-parenting and + self-supersession are still rejected in the `init` block. Cycles longer than one need the other + runs and stay with the store that walks the chains. **Privacy is + a contract with an enforced floor.** `ExtractionActorRef`, `ExtractionRequestRef`, `ExtractionSessionRef`, `ExtractionPersonalizationRef`, `ExtractionDeploymentRef`, `ExtractionExperimentRef` and `ExtractionCohortRef` are all `ExtractionOpaqueRef`: host-minted tokens DICE compares and never @@ -1257,13 +1266,15 @@ and the consumer PRs that deliver it). and a migration later. **One cap rule**: every bound is a named constant on `ExtractionRunLimits`, checked in the `init` block of the type that owns the value, and an over-long value is rejected rather than truncated, because a shortened identifier is a different identifier. Identifiers cap - at 256 characters, source keys at 1024 (they come from `SourceLocator.key()` and can hold a long - URL), the one free-text failure detail at 512, and the three collections at 256 source revisions, - 1024 invocation records and 64 failures. The failure detail is the single exception to rejection - and only on the way in: its factories clip it, the constructor still rejects. `SourceRevisionRef` - predates the rule and validates non-blank only, so `ExtractionRun` applies the bound where it - stores one; moving the check onto that type is follow-up work. `ContextId.value` is the one - string a run stores that the rule does not cover — `ContextId` is a DICE-wide type owned by the + at 256 characters, the one free-text failure detail at 512, and the three collections at 256 + source revisions, 1024 invocation records and 64 failures. The failure detail is the single + exception to rejection and only on the way in: its factories clip it, the constructor still + rejects. `sourceKey` and `sourceRevision` carry no length bound on `ExtractionRun`: their one + bound lives on `SourceRevisionRef`, the type that owns those strings, which checks both halves + against `SourceIdentityBounds` at construction (`docs/design/source-revisions.md`). A run accepts + whatever that type accepts and adds no cap of its own, so a revision a query worked with can + never fail on the way into run recording. + One further string sits outside the rule too: `ContextId.value` — `ContextId` is a DICE-wide type owned by the agent framework — which matters because the tenant is half of `ExtractionRunKey`, so the store key is bounded on one side only. Two bounds are enforced away from the field they protect: `plan(count)` checks the invocation limit against the count before allocating anything, so a diff --git a/dice/src/main/kotlin/com/embabel/dice/common/ConversationAnalysisRequestEvent.kt b/dice/src/main/kotlin/com/embabel/dice/common/ConversationAnalysisRequestEvent.kt index fc3d9c21..1d985d62 100644 --- a/dice/src/main/kotlin/com/embabel/dice/common/ConversationAnalysisRequestEvent.kt +++ b/dice/src/main/kotlin/com/embabel/dice/common/ConversationAnalysisRequestEvent.kt @@ -21,6 +21,7 @@ import com.embabel.chat.Message import com.embabel.dice.incremental.ConversationSource import com.embabel.dice.incremental.IncrementalSource import com.embabel.dice.proposition.extraction.ExtractionContentProfileRef +import com.embabel.dice.proposition.extraction.ExtractionRunRef import com.embabel.dice.provenance.SourceLocator import com.embabel.dice.provenance.SourceRevisionRef @@ -31,13 +32,11 @@ import com.embabel.dice.provenance.SourceRevisionRef * The three-argument constructor is the one that has always existed and carries no * provenance. A publisher that has a typed source for the conversation — a thread in a * chat system, a transcript file — uses the longer constructor to say so, and the same - * constructor takes an extraction content [profile]. EXPERIMENTAL; see [ExtractionContentProfileRef] - * for what carrying it means and does not mean. + * constructor takes an extraction content profile and a run reference. * - * [sourceLocator] is nullable there because profile and source provenance are two independent - * dimensions: a publisher can name a profile for a conversation it has no typed source for. - * Within source provenance, [sourceRevision] rides on [sourceLocator] — it names a version of - * that source, so it needs one. + * [sourceLocator] is nullable there because the four things are independent: a publisher can + * name a profile for a conversation it has no typed source for. Only the revision is coupled, + * and to the locator alone — it names a version of that source, so it needs one. */ class ConversationAnalysisRequestEvent( source: Any, @@ -51,6 +50,8 @@ class ConversationAnalysisRequestEvent( private var eventProfile: ExtractionContentProfileRef? = null + private var eventCurrentRun: ExtractionRunRef? = null + @JvmOverloads constructor( source: Any, @@ -59,10 +60,12 @@ class ConversationAnalysisRequestEvent( sourceLocator: SourceLocator?, sourceRevision: SourceRevisionRef? = null, profile: ExtractionContentProfileRef? = null, + currentRun: ExtractionRunRef? = null, ) : this(source, user, conversation) { eventSourceLocator = sourceLocator eventSourceRevision = sourceRevision eventProfile = profile + eventCurrentRun = currentRun } override fun incrementalSource(): IncrementalSource = @@ -73,4 +76,6 @@ class ConversationAnalysisRequestEvent( override fun sourceRevision(): SourceRevisionRef? = eventSourceRevision override fun profile(): ExtractionContentProfileRef? = eventProfile + + override fun currentRun(): ExtractionRunRef? = eventCurrentRun } diff --git a/dice/src/main/kotlin/com/embabel/dice/common/SourceAnalysisContext.kt b/dice/src/main/kotlin/com/embabel/dice/common/SourceAnalysisContext.kt index 322ad240..bb9174a3 100644 --- a/dice/src/main/kotlin/com/embabel/dice/common/SourceAnalysisContext.kt +++ b/dice/src/main/kotlin/com/embabel/dice/common/SourceAnalysisContext.kt @@ -21,6 +21,7 @@ import com.embabel.dice.provenance.SourceLocator import com.embabel.dice.provenance.SourceRevisionRef import com.embabel.dice.proposition.extraction.ExtractionContentProfileRef import com.embabel.dice.proposition.extraction.ExtractionPerspective +import com.embabel.dice.proposition.extraction.ExtractionRunRef /** * Base context for analyzing sources. @@ -45,6 +46,9 @@ import com.embabel.dice.proposition.extraction.ExtractionPerspective * credential, and no DICE code reads policy out of it. The host authorizes the profile and binds * it to whatever it means. `null` (the default) is the whole of the existing behaviour. * Independent of [perspective], [schema] and [contextId]: setting one never constrains another. + * @param currentRun optional reference to the extraction run this analysis belongs to. + * EXPERIMENTAL. Identity only — DICE #67 brings the durable run this reference will key. + * `null` (the default) means the analysis is attributed to no run, which is every caller today. * @param mintNewEntities whether a mention the resolver could NOT match to an existing entity may * be persisted as a NEW entity node. Default FALSE: unresolved mentions stay unresolved (the * proposition is still persisted; its mention simply carries no resolvedId), so extraction never @@ -72,6 +76,7 @@ data class SourceAnalysisContext @JvmOverloads constructor( val mintedEntityProperties: Map = emptyMap(), val sourceRevision: SourceRevisionRef? = null, val profile: ExtractionContentProfileRef? = null, + val currentRun: ExtractionRunRef? = null, ) { init { @@ -83,10 +88,10 @@ data class SourceAnalysisContext @JvmOverloads constructor( "sourceRevision source key must match sourceLocator source key" } } - // [profile] is checked against nothing else here, deliberately. A revision has to name - // the source it was read from, which is why it is coupled to [sourceLocator]. A profile - // is independent of every other field, and validating it against one would invent a - // relationship the contract doesn't have. + // [profile] and [currentRun] are checked against nothing else here, deliberately. A + // revision has to name the source the run is reading, which is why it is coupled to + // [sourceLocator]. A profile and a run reference are independent of every other field, + // and validating them against one would invent a relationship the contract doesn't have. } companion object { @@ -168,6 +173,13 @@ data class SourceAnalysisContext @JvmOverloads constructor( fun withProfile(profile: ExtractionContentProfileRef): SourceAnalysisContext = copy(profile = profile) + /** + * Returns a copy that says this analysis belongs to the given extraction run. EXPERIMENTAL. + * Changes no other field and no extraction behaviour — see [currentRun]. + */ + fun withCurrentRun(currentRun: ExtractionRunRef): SourceAnalysisContext = + copy(currentRun = currentRun) + /** * Returns a copy allowing (or forbidding) this analysis to persist NEW entities * for mentions the resolver could not match. See [mintNewEntities]. diff --git a/dice/src/main/kotlin/com/embabel/dice/common/SourceAnalysisRequestEvent.kt b/dice/src/main/kotlin/com/embabel/dice/common/SourceAnalysisRequestEvent.kt index 46d37cd1..fd427bf7 100644 --- a/dice/src/main/kotlin/com/embabel/dice/common/SourceAnalysisRequestEvent.kt +++ b/dice/src/main/kotlin/com/embabel/dice/common/SourceAnalysisRequestEvent.kt @@ -19,6 +19,7 @@ import com.embabel.agent.rag.model.NamedEntity import com.embabel.chat.Message import com.embabel.dice.incremental.IncrementalSource import com.embabel.dice.proposition.extraction.ExtractionContentProfileRef +import com.embabel.dice.proposition.extraction.ExtractionRunRef import com.embabel.dice.provenance.SourceLocator import com.embabel.dice.provenance.SourceRevisionRef import org.springframework.context.ApplicationEvent @@ -34,9 +35,9 @@ import org.springframework.context.ApplicationEvent * grounds propositions exactly the way a direct `rememberText` call carrying a request does. Both * default to null, so an existing subclass carries no provenance and behaves as it always did. * - * [profile] works the same way and reaches the same context through the same call, so an async - * publisher can attribute its extraction to a content profile without the listener growing a - * second code path. It also defaults to null. + * [profile] and [currentRun] work the same way and reach the same context through the same + * call, so an async publisher can attribute its extraction to a content profile and a run + * without the listener growing a second code path. Both also default to null. */ abstract class SourceAnalysisRequestEvent( source: Any, @@ -62,4 +63,10 @@ abstract class SourceAnalysisRequestEvent( * publisher has one. EXPERIMENTAL. DICE carries it and routes nothing on it. */ open fun profile(): ExtractionContentProfileRef? = null + + /** + * The extraction run this event's analysis belongs to, when the publisher is running one. + * EXPERIMENTAL. Identity only — nothing is stored under it until DICE #67 lands. + */ + open fun currentRun(): ExtractionRunRef? = null } diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRequest.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRequest.kt index a5e5a74e..f76971c4 100644 --- a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRequest.kt +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRequest.kt @@ -23,20 +23,23 @@ import org.jetbrains.annotations.ApiStatus * What a caller wants to say about one extraction, on top of the text and the user it belongs to. * * Extraction keeps learning about new dimensions — where the material came from, which version of - * it, which content policy it runs under — and each one would otherwise mean another argument on - * [IncrementalPropositionExtraction.rememberText] and another overload to keep the old shape - * callable. They travel here together, so the next dimension is a field on this type and the - * entry-point signatures stay put. A host that overrides an entry point keeps compiling when one - * is added, and sees the new value without touching its override. + * it, which content policy it runs under, which run it belongs to — and each one would otherwise + * mean another argument on [IncrementalPropositionExtraction.rememberText] and another overload to + * keep the old shape callable. They travel here together, so the next dimension is a field on this + * type and the entry-point signatures stay put. A host that overrides an entry point keeps + * compiling when one is added, and sees the new value without touching its override. [currentRun] + * is the first field to arrive that way: it landed with no signature change anywhere. * * Everything is optional. An empty request — [NONE], or `ExtractionRequest()` — asks for the * extraction DICE has always done. * * A [sourceRevision] needs a [sourceLocator] whose key it matches, because a revision names one * version of one specific source. That pairing is checked while the request is being built, so a - * caller finds out about a mismatch before extraction reads a byte. [profile] is checked against - * nothing: it is independent of where the material came from, and coupling it to the other two - * would invent a relationship the contract does not have. + * caller finds out about a mismatch before extraction reads a byte. [profile] and [currentRun] are + * checked against nothing: each is independent of where the material came from, and coupling them + * to the locator and the revision would invent a relationship the contract does not have. DICE + * also never checks that a [currentRun] names a run that exists — the host mints run ids and the + * run store is the only thing that could answer the question. * * EXPERIMENTAL, for as long as [ExtractionContentProfileRef] is. * @@ -49,12 +52,15 @@ import org.jetbrains.annotations.ApiStatus * @property profile the host's content-profile identity for this extraction. DICE carries it and * does nothing else with it: no provider, model, or credential is chosen from it, and extraction * runs exactly as it would without one. + * @property currentRun the extraction run this call belongs to. Identity only: DICE puts it on the + * analysis context and stores nothing under it until the run store lands (DICE #67). */ @ApiStatus.Experimental data class ExtractionRequest @JvmOverloads constructor( val sourceLocator: SourceLocator? = null, val sourceRevision: SourceRevisionRef? = null, val profile: ExtractionContentProfileRef? = null, + val currentRun: ExtractionRunRef? = null, ) { init { @@ -95,6 +101,13 @@ data class ExtractionRequest @JvmOverloads constructor( fun withProfile(profile: ExtractionContentProfileRef): ExtractionRequest = copy(profile = profile) + /** + * Returns a copy that belongs to the given extraction run. EXPERIMENTAL. Changes no other + * field and no extraction behaviour. + */ + fun withCurrentRun(currentRun: ExtractionRunRef): ExtractionRequest = + copy(currentRun = currentRun) + companion object { /** diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRun.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRun.kt index be17b595..61ae08df 100644 --- a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRun.kt +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRun.kt @@ -148,18 +148,10 @@ class ExtractionRun @JvmOverloads constructor( require(this.sourceRevisions.distinct().size == this.sourceRevisions.size) { "sourceRevisions must be distinct; a run reads each source revision once" } - // SourceRevisionRef predates the run model's cap rule and validates non-blank only, so the - // bound is applied where the run stores it. Moving it onto the type is a follow-up. - this.sourceRevisions.forEach { revision -> - require(revision.sourceKey.length <= ExtractionRunLimits.MAX_SOURCE_KEY_LENGTH) { - "sourceKey must be at most ${ExtractionRunLimits.MAX_SOURCE_KEY_LENGTH} characters, " + - "was ${revision.sourceKey.length}" - } - require(revision.sourceRevision.length <= ExtractionRunLimits.MAX_IDENTIFIER_LENGTH) { - "sourceRevision must be at most ${ExtractionRunLimits.MAX_IDENTIFIER_LENGTH} characters, " + - "was ${revision.sourceRevision.length}" - } - } + // sourceKey and sourceRevision carry no length check here. Their one bound lives on + // SourceRevisionRef, the type that owns those strings (docs/design/source-revisions.md), + // and a run accepts whatever that type accepts. Only what belongs to this type — how many + // revisions, and that they are distinct — is checked here. require(this.invocations.size <= ExtractionRunLimits.MAX_INVOCATIONS) { "a run may record at most ${ExtractionRunLimits.MAX_INVOCATIONS} invocation records, " + diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLimits.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLimits.kt index 14cbc98d..12ede88d 100644 --- a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLimits.kt +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLimits.kt @@ -33,9 +33,12 @@ import org.jetbrains.annotations.ApiStatus * Lengths count UTF-16 chars (`String.length`), so a 256-char identifier can be around 1 KB of * UTF-8. The bound is there to keep a run header finite. * - * `SourceRevisionRef` predates this rule and validates non-blank only, so [ExtractionRun] applies - * [MAX_SOURCE_KEY_LENGTH] and [MAX_IDENTIFIER_LENGTH] to the revisions it stores. Moving those - * checks into `SourceRevisionRef` is a follow-up in the module that owns it. + * `sourceKey` and `sourceRevision` are the one pair of strings this rule does not cover, and there + * is no constant here for either. Their bound already exists where it belongs: on + * `SourceRevisionRef`, the type that owns those values, which checks both halves against + * `SourceIdentityBounds` at construction — see its KDoc and `docs/design/source-revisions.md`. + * [ExtractionRun] holds a revision to that same contract and adds no cap of its own, so a revision + * that named a source successfully elsewhere always goes on to be recordable. * * One string a run stores is outside the rule and stays outside it: `ContextId.value`, which * `ExtractionRun` holds as its tenant and validates non-blank only. `ContextId` is a DICE-wide @@ -53,12 +56,6 @@ object ExtractionRunLimits { */ const val MAX_IDENTIFIER_LENGTH: Int = 256 - /** - * Longest source key a run stores. Source keys come out of `SourceLocator.key()` and can - * legitimately hold a long URL, so they get more room than an identifier a host mints itself. - */ - const val MAX_SOURCE_KEY_LENGTH: Int = 1024 - /** * Longest failure detail a run stores. Long enough for a classified one-line explanation, * short enough that a run with the full [MAX_FAILURES] of them stays small. diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLineage.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLineage.kt index de51e2bd..b65b7854 100644 --- a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLineage.kt +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLineage.kt @@ -33,14 +33,27 @@ import org.jetbrains.annotations.ApiStatus * carries an optional `root` alongside the immediate parent, so consumers do not have to walk — * and deep pass-and-retry chains are exactly where walking hurts. * - * The root only stays true if it is never recomputed later, so it is fixed at construction. Use - * [root] and [childOf] and the arithmetic is done for you; the constructor rejects the - * combinations that would make the field a lie. + * **[root] and [childOf] are the only public way to mint one.** The constructor is private, and + * `copy()` follows its visibility too (`@ConsistentCopyVisibility` on this class), so neither can + * hand a caller an independently-set root. [root] sets [rootRunRef] to the run's own ref, since it + * takes no parent at all; [childOf] derives it from the actual parent, the full + * [ExtractionRunLineage] it is handed, so the parent's own record supplies the root. Either way, a root + * that disagrees with the parent chain has no public constructor parameter to arrive through. A + * future store slice reconstructing a lineage from stored fields has to go through [childOf] with + * the parent's own lineage in hand, the same way; re-assembling `rootRunRef` and `parentRunRef` + * from separate columns is the shortcut this closes off. * - * **What is checked here, and what is not.** The constructor rejects self-reference on both axes - * and rejects a root that contradicts the presence of a parent. It cannot see a cycle of length - * two or more, because a value type holds one run and cycle detection needs the other runs. The - * store that walks these chains is where bounded, cycle-safe traversal lives. + * **This closes the public API, and nothing wider.** Kotlin reflection can still call the private + * constructor directly and hand it a root that contradicts the parent it names — + * `ExtractionRunLineageTest` demonstrates the call. Jackson's Kotlin module resolves a data class's + * primary constructor the same reflective way, so a deserializer reading this type from JSON would + * reach the same gap. Nothing serializes an `ExtractionRunLineage` today; this is a residual for + * whoever builds that wiring, unrelated to anything that exists yet. + * + * **What is checked here, and what is not.** The constructor rejects self-reference on both axes. + * It cannot see a cycle of length two or more, because a value type holds one run and cycle + * detection needs the other runs. The store that walks these chains is where bounded, cycle-safe + * traversal lives. * * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. * @@ -52,7 +65,8 @@ import org.jetbrains.annotations.ApiStatus * @property passIndex Which pass over the material this is, counting from zero */ @ApiStatus.Experimental -data class ExtractionRunLineage @JvmOverloads constructor( +@ConsistentCopyVisibility +data class ExtractionRunLineage private constructor( val runRef: ExtractionRunRef, val rootRunRef: ExtractionRunRef, val parentRunRef: ExtractionRunRef? = null, @@ -116,5 +130,38 @@ data class ExtractionRunLineage @JvmOverloads constructor( supersedesRunRef = supersedesRunRef, passIndex = passIndex, ) + + /** + * Rebuilds a lineage a store previously wrote, from the columns it wrote it to. + * + * [root] and [childOf] are the only ways to mint a new lineage, and they exist so a root + * cannot be stated independently of the parent it belongs to. A store reading a row back + * has the opposite problem: it holds five stored values and no parent lineage to derive + * anything from, and the values it holds are ones this type already accepted on the way in. + * + * The init guards still run, so a row that has become self-referential is rejected here. + * What this call trusts is the relationship between root and parent, and a backend is + * expected to have its own check on that: `DrivineExtractionRunStore` stores a lineage key + * and compares it with the key of what it just rebuilt, so a row edited underneath it fails + * to load, and never loads wrong. + * + * For store backends. Application code mints through [root] and [childOf]. + */ + @ApiStatus.Internal + @JvmStatic + @JvmOverloads + fun fromStoredFields( + runRef: ExtractionRunRef, + rootRunRef: ExtractionRunRef, + parentRunRef: ExtractionRunRef? = null, + supersedesRunRef: ExtractionRunRef? = null, + passIndex: Int = 0, + ): ExtractionRunLineage = ExtractionRunLineage( + runRef = runRef, + rootRunRef = rootRunRef, + parentRunRef = parentRunRef, + supersedesRunRef = supersedesRunRef, + passIndex = passIndex, + ) } } diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/IncrementalPropositionExtraction.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/IncrementalPropositionExtraction.kt index 700032c0..434dc7f8 100644 --- a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/IncrementalPropositionExtraction.kt +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/IncrementalPropositionExtraction.kt @@ -178,7 +178,8 @@ open class IncrementalPropositionExtraction @JvmOverloads constructor( /** * Extract propositions from a file via Tika, on the terms the [request] sets — the source it - * was read from, the revision of that source, the content profile it runs under. + * was read from, the revision of that source, the content profile it runs under, the + * extraction run it belongs to. * * A [request] that carries nothing hands straight back to the three-argument form, so a call * that looks like a pre-request call also dispatches like one. @@ -274,7 +275,8 @@ open class IncrementalPropositionExtraction @JvmOverloads constructor( /** * Extract propositions from raw text on the terms the [request] sets — the source it was read - * from, the revision of that source, the content profile it runs under. + * from, the revision of that source, the content profile it runs under, the extraction run it + * belongs to. * * Every other entry point funnels here, so this is the one method to override to see every * call. It is also where a new extraction dimension shows up: it arrives as a field on @@ -356,6 +358,7 @@ open class IncrementalPropositionExtraction @JvmOverloads constructor( sourceLocator = event.sourceLocator(), sourceRevision = event.sourceRevision(), profile = event.profile(), + currentRun = event.currentRun(), ), ) logger.info( @@ -452,9 +455,11 @@ open class IncrementalPropositionExtraction @JvmOverloads constructor( // context and the request always agree about a call. request.sourceLocator?.let { ctx = ctx.withSourceLocator(it) } request.sourceRevision?.let { ctx = ctx.withSourceRevision(it) } - // Carried, never consulted. Nothing downstream of here reads it — that is what "DICE - // holds profile identity and the host binds policy" means in code. + // Carried, never consulted. Nothing downstream of here reads either one — that is what + // "DICE holds profile identity and the host binds policy" means in code, and the run + // reference is identity only until the run store lands. request.profile?.let { ctx = ctx.withProfile(it) } + request.currentRun?.let { ctx = ctx.withCurrentRun(it) } return ctx } diff --git a/dice/src/main/kotlin/com/embabel/dice/provenance/SourceRevisionRef.kt b/dice/src/main/kotlin/com/embabel/dice/provenance/SourceRevisionRef.kt index 0a57f9cb..094a5a50 100644 --- a/dice/src/main/kotlin/com/embabel/dice/provenance/SourceRevisionRef.kt +++ b/dice/src/main/kotlin/com/embabel/dice/provenance/SourceRevisionRef.kt @@ -20,8 +20,16 @@ import org.jetbrains.annotations.ApiStatus /** * Identifies an opaque revision of a source. * + * A revision is opaque to DICE, exactly as `docs/design/source-revisions.md` describes it: a + * provider-defined value DICE stores and compares for exact equality, and never parses. + * * Both halves are checked against [SourceIdentityBounds] on construction, so a value too long to - * store or index is refused here, before it can reach a query or a write. + * store or index is refused here, before it can reach a query or a write. Those two ceilings are + * the only limit either field has, and they live on this type because this type owns the value. + * Everything downstream that holds a `SourceRevisionRef` — a revision query, a REST request, + * evidence carried in from somewhere else, run recording — inherits the same guarantee and adds + * no cap of its own. So length is never the reason a value that named a source successfully in + * one place goes on to be refused in another. * * @property sourceKey Canonical source identity produced by [SourceLocator.key] * @property sourceRevision Provider-defined opaque revision value diff --git a/dice/src/test/java/com/embabel/dice/ExtractionProfileJavaInteropTest.java b/dice/src/test/java/com/embabel/dice/ExtractionProfileJavaInteropTest.java index 57cb31d5..70ec4bd7 100644 --- a/dice/src/test/java/com/embabel/dice/ExtractionProfileJavaInteropTest.java +++ b/dice/src/test/java/com/embabel/dice/ExtractionProfileJavaInteropTest.java @@ -27,6 +27,7 @@ import com.embabel.dice.proposition.extraction.ExtractionContentProfileRef; import com.embabel.dice.proposition.extraction.ExtractionPerspective; import com.embabel.dice.proposition.extraction.ExtractionRequest; +import com.embabel.dice.proposition.extraction.ExtractionRunRef; import com.embabel.dice.proposition.extraction.IncrementalPropositionExtraction; import com.embabel.dice.provenance.ContentAddressedLocator; import com.embabel.dice.provenance.SourceLocator; @@ -45,9 +46,10 @@ import static org.junit.jupiter.api.Assertions.assertThrows; /** - * Java's view of the profile contract, with and without a profile present. A profile travels on an + * Java's view of the profile and run contracts, with and without either present. Both travel on an * {@link ExtractionRequest}, so the remember entry points keep every descriptor a Java caller could - * already have compiled against and each name gains exactly one more, taking the request. + * already have compiled against and each name gains exactly one more, taking the request. Adding + * the run reference added no descriptor at all — it is a field on the request. */ class ExtractionProfileJavaInteropTest { @@ -59,7 +61,7 @@ private SourceAnalysisContext context() { } @Test - void javaCallersBuildAndReadProfileValues() { + void javaCallersBuildAndReadProfileAndRunValues() { ExtractionContentProfileRef profile = new ExtractionContentProfileRef("house-style", "v1"); assertEquals("house-style", profile.getName()); assertEquals("v1", profile.getVersion()); @@ -69,19 +71,28 @@ void javaCallersBuildAndReadProfileValues() { IllegalArgumentException.class, () -> new ExtractionContentProfileRef(" ", "v1") ); + + ExtractionRunRef run = new ExtractionRunRef("run-1"); + assertEquals("run-1", run.getRunId()); + assertEquals(run, new ExtractionRunRef("run-1")); + assertThrows(IllegalArgumentException.class, () -> new ExtractionRunRef("")); } @Test void javaBuiltContextsCarryNoProfileUnlessAsked() { SourceAnalysisContext absent = context(); assertNull(absent.getProfile()); + assertNull(absent.getCurrentRun()); ExtractionContentProfileRef profile = new ExtractionContentProfileRef("house-style", "v1"); - SourceAnalysisContext present = absent.withProfile(profile); + ExtractionRunRef run = new ExtractionRunRef("run-1"); + SourceAnalysisContext present = absent.withProfile(profile).withCurrentRun(run); assertSame(profile, present.getProfile()); + assertSame(run, present.getCurrentRun()); // The copy is a copy: the original is untouched. assertNull(absent.getProfile()); + assertNull(absent.getCurrentRun()); } @Test @@ -110,8 +121,9 @@ void retainsEveryLegacyRememberDescriptorAndAddsRequestAwareOnes() throws Except InputStream.class, String.class, NamedEntity.class, ExtractionRequest.class ); - // A profile reaches extraction on the request, so no entry point takes one directly and - // none lands where a caller filling every legacy argument already is. + // A profile and a run both reach extraction on the request, so no entry point takes + // either one directly and neither lands where a caller filling every legacy argument + // already is. assertThrows( NoSuchMethodException.class, () -> IncrementalPropositionExtraction.class.getMethod( @@ -128,6 +140,15 @@ void retainsEveryLegacyRememberDescriptorAndAddsRequestAwareOnes() throws Except ExtractionPerspective.class, Boolean.class, ExtractionContentProfileRef.class ) ); + assertThrows( + NoSuchMethodException.class, + () -> IncrementalPropositionExtraction.class.getMethod( + "rememberText", + String.class, String.class, NamedEntity.class, List.class, + ExtractionPerspective.class, Boolean.class, + ExtractionContentProfileRef.class, ExtractionRunRef.class + ) + ); assertThrows( NoSuchMethodException.class, () -> IncrementalPropositionExtraction.class.getMethod( @@ -136,30 +157,71 @@ void retainsEveryLegacyRememberDescriptorAndAddsRequestAwareOnes() throws Except ExtractionContentProfileRef.class ) ); + assertThrows( + NoSuchMethodException.class, + () -> IncrementalPropositionExtraction.class.getMethod( + "rememberFile", + InputStream.class, String.class, NamedEntity.class, + ExtractionRunRef.class + ) + ); + assertThrows( + NoSuchMethodException.class, + () -> IncrementalPropositionExtraction.class.getMethod( + "rememberFile", + InputStream.class, String.class, NamedEntity.class, + ExtractionContentProfileRef.class, ExtractionRunRef.class + ) + ); + + // The `...FromSource` names are gone entirely: a typed source, its revision, the profile + // and the run all travel on the request, so there is nothing left for a second family of + // entry points to carry. No descriptor under either name survives, at any arity. + assertFalse( + java.util.Arrays.stream(IncrementalPropositionExtraction.class.getMethods()) + .anyMatch(method -> method.getName().endsWith("FromSource")) + ); } @Test - void javaCallersBuildAndReadProfileBearingRequests() throws Exception { + void javaCallersBuildAndReadProfileAndRunBearingRequests() throws Exception { ExtractionContentProfileRef profile = new ExtractionContentProfileRef("house-style", "v1"); + ExtractionRunRef run = new ExtractionRunRef("run-1"); SourceLocator locator = new ContentAddressedLocator("java-profile-source"); SourceRevisionRef revision = new SourceRevisionRef(locator.key(), "r1"); + // `@JvmOverloads` publishes the shorter descriptors too, so a Java caller that compiled + // against the three-field request keeps compiling now the run field exists. ExtractionRequest.class.getConstructor( SourceLocator.class, SourceRevisionRef.class, ExtractionContentProfileRef.class ); + ExtractionRequest.class.getConstructor( + SourceLocator.class, SourceRevisionRef.class, ExtractionContentProfileRef.class, + ExtractionRunRef.class + ); - // A profile needs no source of its own: the two dimensions stay independent on a request. + // A profile needs no source of its own: the dimensions stay independent on a request. ExtractionRequest profileOnly = new ExtractionRequest(null, null, profile); assertSame(profile, profileOnly.getProfile()); assertNull(profileOnly.getSourceLocator()); + assertNull(profileOnly.getCurrentRun()); + + // Neither does a run. + ExtractionRequest runOnly = new ExtractionRequest(null, null, null, run); + assertSame(run, runOnly.getCurrentRun()); + assertNull(runOnly.getSourceLocator()); + assertNull(runOnly.getProfile()); - ExtractionRequest everything = new ExtractionRequest(locator, revision, profile); + ExtractionRequest everything = new ExtractionRequest(locator, revision, profile, run); assertSame(profile, everything.getProfile()); assertSame(revision, everything.getSourceRevision()); + assertSame(run, everything.getCurrentRun()); - // The copy helper is a copy: the request it was called on is untouched. + // The copy helpers are copies: the request they were called on is untouched. assertSame(profile, ExtractionRequest.NONE.withProfile(profile).getProfile()); assertNull(ExtractionRequest.NONE.getProfile()); + assertSame(run, ExtractionRequest.NONE.withCurrentRun(run).getCurrentRun()); + assertNull(ExtractionRequest.NONE.getCurrentRun()); } @Test @@ -168,10 +230,13 @@ void legacyAndProfileAwareJavaEventSubclassesUseTheBaseConstructor() { LegacyJavaEvent legacy = new LegacyJavaEvent(this, user); assertNull(legacy.profile()); + assertNull(legacy.currentRun()); ExtractionContentProfileRef profile = new ExtractionContentProfileRef("house-style", "v1"); - ProfileAwareJavaEvent profileAware = new ProfileAwareJavaEvent(this, user, profile); + ExtractionRunRef run = new ExtractionRunRef("run-1"); + ProfileAwareJavaEvent profileAware = new ProfileAwareJavaEvent(this, user, profile, run); assertSame(profile, profileAware.profile()); + assertSame(run, profileAware.currentRun()); // A subclass that only knows about profiles still carries no source provenance. assertNull(profileAware.sourceLocator()); assertNull(profileAware.sourceRevision()); @@ -196,37 +261,54 @@ void conversationEventKeepsItsFiveArgumentConstructorAndGainsProfileAwareOnes() Object.class, NamedEntity.class, Conversation.class, SourceLocator.class, SourceRevisionRef.class, ExtractionContentProfileRef.class ); + ConversationAnalysisRequestEvent.class.getConstructor( + Object.class, NamedEntity.class, Conversation.class, SourceLocator.class, + SourceRevisionRef.class, ExtractionContentProfileRef.class, ExtractionRunRef.class + ); NamedEntity user = org.mockito.Mockito.mock(NamedEntity.class); Conversation conversation = org.mockito.Mockito.mock(Conversation.class); SourceLocator locator = new ContentAddressedLocator("java-conversation"); SourceRevisionRef revision = new SourceRevisionRef(locator.key(), "r1"); ExtractionContentProfileRef profile = new ExtractionContentProfileRef("house-style", "v1"); + ExtractionRunRef run = new ExtractionRunRef("run-1"); ConversationAnalysisRequestEvent locatorOnly = new ConversationAnalysisRequestEvent(this, user, conversation, locator); assertSame(locator, locatorOnly.sourceLocator()); assertNull(locatorOnly.sourceRevision()); assertNull(locatorOnly.profile()); + assertNull(locatorOnly.currentRun()); ConversationAnalysisRequestEvent legacy = new ConversationAnalysisRequestEvent(this, user, conversation, locator, revision); assertSame(locator, legacy.sourceLocator()); assertSame(revision, legacy.sourceRevision()); assertNull(legacy.profile()); + assertNull(legacy.currentRun()); + // The 6-argument form a profile-aware Java caller already compiled against, unchanged: it + // still names a profile and now leaves the run defaulted. ConversationAnalysisRequestEvent profiled = new ConversationAnalysisRequestEvent( this, user, conversation, locator, revision, profile ); assertSame(profile, profiled.profile()); + assertNull(profiled.currentRun()); + + ConversationAnalysisRequestEvent profiledAndRun = new ConversationAnalysisRequestEvent( + this, user, conversation, locator, revision, profile, run + ); + assertSame(profile, profiledAndRun.profile()); + assertSame(run, profiledAndRun.currentRun()); - // A profile without a typed source: the locator argument is nullable because the two + // A profile and a run without a typed source: the locator argument is nullable because the // dimensions are independent. ConversationAnalysisRequestEvent profileOnly = new ConversationAnalysisRequestEvent( - this, user, conversation, null, null, profile + this, user, conversation, null, null, profile, run ); assertNull(profileOnly.sourceLocator()); assertSame(profile, profileOnly.profile()); + assertSame(run, profileOnly.currentRun()); } @Test @@ -317,7 +399,8 @@ private static void compileJavaSourceCalls( NamedEntity user, SourceLocator locator, SourceRevisionRef revision, - ExtractionContentProfileRef profile + ExtractionContentProfileRef profile, + ExtractionRunRef run ) { extraction.rememberText("legacy", "legacy-id", user); extraction.rememberText("legacy", "legacy-id", user, List.of(), null, null); @@ -334,6 +417,31 @@ private static void compileJavaSourceCalls( extraction.rememberFile( input, "source.txt", user, new ExtractionRequest(locator, revision, profile) ); + // A run travels the same way, on the same two entry points. Where the pre-request surface + // needed a `rememberTextFromSource` and a `rememberFileFromSource` to carry a typed source, + // the request carries the source, its revision, the profile and the run together. + extraction.rememberText( + "run", "run-id", user, List.of(), null, null, + new ExtractionRequest(null, null, null, run) + ); + extraction.rememberFile(input, "run.txt", user, new ExtractionRequest(null, null, null, run)); + extraction.rememberText( + "everything", "everything-id", user, List.of(), null, null, + new ExtractionRequest(locator, revision, profile, run) + ); + extraction.rememberFile( + input, "everything.txt", user, new ExtractionRequest(locator, revision, profile, run) + ); + // The same request assembled with the copy helpers, which is how a Java caller that starts + // from NONE builds one. + extraction.rememberText( + "built", "built-id", user, List.of(), null, null, + ExtractionRequest.NONE + .withSourceLocator(locator) + .withSourceRevision(revision) + .withProfile(profile) + .withCurrentRun(run) + ); } private static final class LegacyJavaEvent extends SourceAnalysisRequestEvent { @@ -351,14 +459,17 @@ public IncrementalSource incrementalSource() { private static final class ProfileAwareJavaEvent extends SourceAnalysisRequestEvent { private final ExtractionContentProfileRef profile; + private final ExtractionRunRef run; private ProfileAwareJavaEvent( Object source, NamedEntity user, - ExtractionContentProfileRef profile + ExtractionContentProfileRef profile, + ExtractionRunRef run ) { super(source, user); this.profile = profile; + this.run = run; } @Override @@ -370,5 +481,10 @@ public IncrementalSource incrementalSource() { public ExtractionContentProfileRef profile() { return profile; } + + @Override + public ExtractionRunRef currentRun() { + return run; + } } } diff --git a/dice/src/test/kotlin/com/embabel/dice/ExtractionProfileCompatibilityTest.kt b/dice/src/test/kotlin/com/embabel/dice/ExtractionProfileCompatibilityTest.kt index 9a2d1638..815ae2e3 100644 --- a/dice/src/test/kotlin/com/embabel/dice/ExtractionProfileCompatibilityTest.kt +++ b/dice/src/test/kotlin/com/embabel/dice/ExtractionProfileCompatibilityTest.kt @@ -24,6 +24,7 @@ import com.embabel.dice.common.SourceAnalysisContext import com.embabel.dice.common.resolver.AlwaysCreateEntityResolver import com.embabel.dice.proposition.extraction.ExtractionContentProfileRef import com.embabel.dice.proposition.extraction.ExtractionPerspective +import com.embabel.dice.proposition.extraction.ExtractionRunRef import com.embabel.dice.provenance.ContentAddressedLocator import com.embabel.dice.provenance.SourceLocator import com.embabel.dice.provenance.SourceRevisionRef @@ -43,15 +44,17 @@ class ExtractionProfileCompatibilityTest { private fun context( profile: ExtractionContentProfileRef? = null, + currentRun: ExtractionRunRef? = null, ) = SourceAnalysisContext( schema = DataDictionary.fromClasses("profile-compatibility"), entityResolver = AlwaysCreateEntityResolver, contextId = ContextId("profile-compatibility"), profile = profile, + currentRun = currentRun, ) @Test - fun `legacy Kotlin source constructors and copy calls see no profile`() { + fun `legacy Kotlin source constructors and copy calls see no profile and no run`() { val legacy = SourceAnalysisContext( schema = DataDictionary.fromClasses("profile-compatibility"), entityResolver = AlwaysCreateEntityResolver, @@ -60,26 +63,31 @@ class ExtractionProfileCompatibilityTest { assertEquals(true, legacy.promptVariables["legacy"]) assertNull(legacy.profile) + assertNull(legacy.currentRun) - // The Java-facing builder is unchanged too, and its result carries none. + // The Java-facing builder is unchanged too, and its result carries neither. val built = SourceAnalysisContext .withContextId("profile-compatibility") .withEntityResolver(AlwaysCreateEntityResolver) .withSchema(DataDictionary.fromClasses("profile-compatibility")) assertNull(built.profile) + assertNull(built.currentRun) } @Test - fun `new Kotlin source constructors and copy calls carry a profile`() { + fun `new Kotlin source constructors and copy calls carry a profile and a run`() { val profile = ExtractionContentProfileRef("house-style", "v3") + val run = ExtractionRunRef("run-42") - val fromConstructor = context(profile = profile) + val fromConstructor = context(profile = profile, currentRun = run) .copy(promptVariables = mapOf("profiled" to true)) assertSame(profile, fromConstructor.profile) + assertSame(run, fromConstructor.currentRun) assertEquals(true, fromConstructor.promptVariables["profiled"]) - val fromHelpers = context().withProfile(profile) + val fromHelpers = context().withProfile(profile).withCurrentRun(run) assertSame(profile, fromHelpers.profile) + assertSame(run, fromHelpers.currentRun) } @Test @@ -95,6 +103,7 @@ class ExtractionProfileCompatibilityTest { perspective = ExtractionPerspective.USER, sourceRevision = revision, profile = ExtractionContentProfileRef("house-style", "v3"), + currentRun = ExtractionRunRef("run-42"), ) assertSame(locator, context.sourceLocator) @@ -121,6 +130,7 @@ class ExtractionProfileCompatibilityTest { Map::class.java, SourceRevisionRef::class.java, ExtractionContentProfileRef::class.java, + ExtractionRunRef::class.java, ) val published = SourceAnalysisContext::class.java.constructors .map { it.parameterTypes.toList() } @@ -133,11 +143,13 @@ class ExtractionProfileCompatibilityTest { "constructor of $arity arguments no longer published", ) } - // 12 is what this slice adds, on the end. - assertTrue( - declared.take(12) + marker in published, - "constructor of 12 arguments was not published", - ) + // 12 and 13 are what this slice adds, on the end. + for (arity in 12..13) { + assertTrue( + declared.take(arity) + marker in published, + "constructor of $arity arguments was not published", + ) + } } @Test @@ -147,11 +159,11 @@ class ExtractionProfileCompatibilityTest { val copyArities = SourceAnalysisContext::class.java.declaredMethods .filter { it.name.startsWith("copy") && !it.name.endsWith("\$default") } .map { it.parameterCount } - assertEquals(listOf(12), copyArities) + assertEquals(listOf(13), copyArities) val componentCount = SourceAnalysisContext::class.java.declaredMethods .count { it.name.startsWith("component") } - assertEquals(12, componentCount) + assertEquals(13, componentCount) } @Test diff --git a/dice/src/test/kotlin/com/embabel/dice/common/ExtractionContextIndependenceTest.kt b/dice/src/test/kotlin/com/embabel/dice/common/ExtractionContextIndependenceTest.kt index d056b430..5568bedd 100644 --- a/dice/src/test/kotlin/com/embabel/dice/common/ExtractionContextIndependenceTest.kt +++ b/dice/src/test/kotlin/com/embabel/dice/common/ExtractionContextIndependenceTest.kt @@ -20,6 +20,7 @@ import com.embabel.agent.core.DataDictionary import com.embabel.dice.common.resolver.AlwaysCreateEntityResolver import com.embabel.dice.proposition.extraction.ExtractionContentProfileRef import com.embabel.dice.proposition.extraction.ExtractionPerspective +import com.embabel.dice.proposition.extraction.ExtractionRunRef import com.embabel.dice.provenance.ContentAddressedLocator import com.embabel.dice.provenance.SourceRevisionRef import org.junit.jupiter.api.Assertions.assertEquals @@ -173,6 +174,7 @@ class ExtractionContextIndependenceTest { mintedEntityProperties = mapOf("owner" to "tenant-one"), sourceRevision = SourceRevisionRef(locator.key(), "r1"), profile = profiles[1], + currentRun = ExtractionRunRef("run-before"), ) // Comparing against copy(...) is a statement about every component at once: the helper @@ -180,6 +182,9 @@ class ExtractionContextIndependenceTest { val profile = ExtractionContentProfileRef("legal-review", "v9") assertEquals(base.copy(profile = profile), base.withProfile(profile)) + val run = ExtractionRunRef("run-after") + assertEquals(base.copy(currentRun = run), base.withCurrentRun(run)) + assertEquals( base.copy(perspective = ExtractionPerspective.ALL), base.withPerspective(ExtractionPerspective.ALL), @@ -224,4 +229,28 @@ class ExtractionContextIndependenceTest { ) } } + + @Test + fun `profile and current run are independent of each other`() { + val profile = profiles[1]!! + val run = ExtractionRunRef("run-7") + + val combinations = listOf( + null to null, + profile to null, + null to run, + profile to run, + ).map { (p, r) -> + SourceAnalysisContext( + schema = schemas[0], + entityResolver = AlwaysCreateEntityResolver, + contextId = tenants[0], + profile = p, + currentRun = r, + ) + } + + assertEquals(listOf(null, profile, null, profile), combinations.map { it.profile }) + assertEquals(listOf(null, null, run, run), combinations.map { it.currentRun }) + } } diff --git a/dice/src/test/kotlin/com/embabel/dice/common/SourceAnalysisRequestEventProfileTest.kt b/dice/src/test/kotlin/com/embabel/dice/common/SourceAnalysisRequestEventProfileTest.kt index 72ee2a70..989a7fe9 100644 --- a/dice/src/test/kotlin/com/embabel/dice/common/SourceAnalysisRequestEventProfileTest.kt +++ b/dice/src/test/kotlin/com/embabel/dice/common/SourceAnalysisRequestEventProfileTest.kt @@ -20,6 +20,7 @@ import com.embabel.chat.Conversation import com.embabel.chat.Message import com.embabel.dice.incremental.IncrementalSource import com.embabel.dice.proposition.extraction.ExtractionContentProfileRef +import com.embabel.dice.proposition.extraction.ExtractionRunRef import com.embabel.dice.provenance.ContentAddressedLocator import com.embabel.dice.provenance.SourceRevisionRef import org.junit.jupiter.api.Assertions.assertEquals @@ -29,25 +30,27 @@ import org.junit.jupiter.api.Test import org.mockito.Mockito.mock /** - * The async publisher's half of the profile contract: the accessor defaults to null, an - * existing subclass is unaffected, and the shipped conversation event carries a profile when - * its longer constructor is used. + * The async publisher's half of the profile contract: both accessors default to null, an + * existing subclass is unaffected, and the shipped conversation event carries a profile and a + * run when its longer constructor is used. */ class SourceAnalysisRequestEventProfileTest { private val profile = ExtractionContentProfileRef("house-style", "v1") + private val run = ExtractionRunRef("run-1") @Test - fun `a subclass written before profiles carries none`() { + fun `a subclass written before profiles carries neither`() { val legacy = LegacyEvent(this, mock(NamedEntity::class.java)) assertNull(legacy.profile()) + assertNull(legacy.currentRun()) assertNull(legacy.sourceLocator()) assertNull(legacy.sourceRevision()) } @Test - fun `the shipped conversation event defaults to no profile`() { + fun `the shipped conversation event defaults to no profile and no run`() { val event = ConversationAnalysisRequestEvent( source = this, user = mock(NamedEntity::class.java), @@ -55,10 +58,11 @@ class SourceAnalysisRequestEventProfileTest { ) assertNull(event.profile()) + assertNull(event.currentRun()) } @Test - fun `the conversation event carries the exact profile it was given`() { + fun `the conversation event carries the exact profile and run it was given`() { val locator = ContentAddressedLocator("event-source") val revision = SourceRevisionRef(locator.key(), "r1") @@ -69,11 +73,13 @@ class SourceAnalysisRequestEventProfileTest { sourceLocator = locator, sourceRevision = revision, profile = profile, + currentRun = run, ) assertSame(locator, event.sourceLocator()) assertSame(revision, event.sourceRevision()) assertSame(profile, event.profile()) + assertSame(run, event.currentRun()) } @Test @@ -89,6 +95,7 @@ class SourceAnalysisRequestEventProfileTest { assertNull(event.sourceLocator()) assertNull(event.sourceRevision()) assertSame(profile, event.profile()) + assertNull(event.currentRun()) } @Test @@ -101,6 +108,7 @@ class SourceAnalysisRequestEventProfileTest { } assertSame(profile, profileOnly.profile()) + assertNull(profileOnly.currentRun()) assertNull(profileOnly.sourceLocator()) } diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionProfileContractTest.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionProfileContractTest.kt index 20e55e98..f2d19b1b 100644 --- a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionProfileContractTest.kt +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionProfileContractTest.kt @@ -22,8 +22,8 @@ import org.assertj.core.api.Assertions.assertThatIllegalArgumentException import org.junit.jupiter.api.Test /** - * What the new reference type promises: stable name-and-version identity for a profile, a - * bounded string, and no interpretation of either component. + * What the two new reference types promise: stable name-and-version identity for a profile, + * an opaque id for a run, bounded strings, and no interpretation of either. */ class ExtractionProfileContractTest { @@ -87,4 +87,58 @@ class ExtractionProfileContractTest { .isThrownBy { ExtractionContentProfileRef(name, version + "v") } .withMessageContaining("version") } + + @Test + fun `run ref preserves an opaque id`() { + val ref = ExtractionRunRef("01J9Z0V1XQ:host/7#a") + + assertThat(ref.runId).isEqualTo("01J9Z0V1XQ:host/7#a") + assertThat(objectMapper.readValue(objectMapper.writeValueAsString(ref))) + .isEqualTo(ref) + } + + @Test + fun `run identity is the id`() { + val run = ExtractionRunRef("run-1") + val same = ExtractionRunRef("run-1") + val other = ExtractionRunRef("run-2") + + assertThat(run).isEqualTo(same) + assertThat(run.hashCode()).isEqualTo(same.hashCode()) + assertThat(run).isNotEqualTo(other) + } + + @Test + fun `run ref rejects a blank id`() { + assertThatIllegalArgumentException() + .isThrownBy { ExtractionRunRef("") } + .withMessageContaining("runId") + assertThatIllegalArgumentException() + .isThrownBy { ExtractionRunRef(" ") } + .withMessageContaining("runId") + } + + @Test + fun `run ref accepts its length cap and rejects one character more`() { + val runId = "r".repeat(ExtractionRunRef.MAX_RUN_ID_LENGTH) + + assertThat(ExtractionRunRef(runId).runId).hasSize(ExtractionRunRef.MAX_RUN_ID_LENGTH) + assertThatIllegalArgumentException() + .isThrownBy { ExtractionRunRef(runId + "r") } + .withMessageContaining("runId") + } + + @Test + fun `the two references are unrelated types`() { + // Nothing converts one into the other and neither derives from the other. A profile says + // what extraction should do; a run says which execution this was. + assertThat( + ExtractionContentProfileRef::class.java.isAssignableFrom(ExtractionRunRef::class.java), + ).isFalse() + assertThat( + ExtractionRunRef::class.java.isAssignableFrom(ExtractionContentProfileRef::class.java), + ).isFalse() + assertThat(ExtractionContentProfileRef("run-1", "run-1")) + .isNotEqualTo(ExtractionRunRef("run-1")) + } } diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunContractTest.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunContractTest.kt index c1fb9a09..2719bd05 100644 --- a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunContractTest.kt +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunContractTest.kt @@ -22,6 +22,7 @@ import com.embabel.dice.proposition.extraction.ExtractionRunFixtures.REVISION_ON import com.embabel.dice.proposition.extraction.ExtractionRunFixtures.REVISION_TWO import com.embabel.dice.proposition.extraction.ExtractionRunFixtures.RUN import com.embabel.dice.proposition.extraction.ExtractionRunFixtures.STARTED_AT +import com.embabel.dice.provenance.SourceIdentityBounds import com.embabel.dice.provenance.SourceRevisionRef import org.assertj.core.api.Assertions.assertThat import org.assertj.core.api.Assertions.assertThatIllegalArgumentException @@ -160,30 +161,35 @@ class ExtractionRunContractTest { } @Test - fun `the run applies the cap rule to source revisions its own type does not check`() { - // SourceRevisionRef predates the rule and validates non-blank only, so the bound is applied - // where the run stores it. - assertThatIllegalArgumentException().isThrownBy { - runWith( - sourceRevisions = listOf( - SourceRevisionRef("u".repeat(ExtractionRunLimits.MAX_SOURCE_KEY_LENGTH + 1), "rev-1"), - ), - ) - }.withMessageContaining("sourceKey") + fun `run recording adds no per-field length cap on a source revision`() { + // Repro for the PR #95 review comment, round 2: a value a query happily works with must + // not blow up on the way into run recording. The bound on a source key and a source + // revision belongs to SourceRevisionRef, the type that owns those strings, and the run + // holds itself to whatever that type accepts. So the test builds the biggest revision the + // owning type will mint and records it: both halves are far past the run's own + // MAX_IDENTIFIER_LENGTH, which proves the run's identifier cap does not reach them. + // (The run still rejects duplicates and more than MAX_SOURCE_REVISIONS of them — see + // `every collection the run stores is bounded` above — this test is about length only.) + val revision = SourceRevisionRef( + "u".repeat(SourceIdentityBounds.MAX_SOURCE_KEY_LENGTH), + "r".repeat(SourceIdentityBounds.MAX_SOURCE_REVISION_LENGTH), + ) + assertThat(revision.sourceKey.length).isGreaterThan(ExtractionRunLimits.MAX_IDENTIFIER_LENGTH) + assertThat(revision.sourceRevision.length).isGreaterThan(ExtractionRunLimits.MAX_IDENTIFIER_LENGTH) - assertThatIllegalArgumentException().isThrownBy { - runWith( - sourceRevisions = listOf( - SourceRevisionRef("uri:doc-a", "r".repeat(ExtractionRunLimits.MAX_IDENTIFIER_LENGTH + 1)), - ), - ) - }.withMessageContaining("sourceRevision") + assertThat(runWith(sourceRevisions = listOf(revision)).sourceRevisions) + .containsExactly(revision) - // A long URL as a source key is exactly why source keys get more room than identifiers. - val longUrlKey = "https://example.test/" + "segment/".repeat(60) + // A long URL as a source key is exactly the kind of value SourceRevisionRef has to carry. + val longUrlKey = "https://example.test/" + "segment/".repeat(200) assertThat(longUrlKey.length).isGreaterThan(ExtractionRunLimits.MAX_IDENTIFIER_LENGTH) assertThat(runWith(sourceRevisions = listOf(SourceRevisionRef(longUrlKey, "rev-1"))).sourceRevisions) .hasSize(1) + + // And there is no constant here for either field, so a cap cannot creep back in on this + // side without someone noticing. + assertThat(ExtractionRunLimits::class.java.declaredFields.map { it.name }) + .doesNotContain("MAX_SOURCE_KEY_LENGTH", "MAX_SOURCE_REVISION_LENGTH") } @Test diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLineageTest.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLineageTest.kt index f1e7ede8..054f2368 100644 --- a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLineageTest.kt +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLineageTest.kt @@ -17,12 +17,19 @@ package com.embabel.dice.proposition.extraction import org.assertj.core.api.Assertions.assertThat import org.assertj.core.api.Assertions.assertThatIllegalArgumentException +import org.assertj.core.api.Assertions.catchThrowable import org.junit.jupiter.api.Test +import java.lang.reflect.InvocationTargetException +import kotlin.reflect.KVisibility +import kotlin.reflect.full.declaredFunctions +import kotlin.reflect.full.primaryConstructor +import kotlin.reflect.jvm.isAccessible /** * The root reference is denormalized, so it is only worth having if it cannot drift: a parentless - * run is its own root, a child takes its parent's root, and the constructor rejects every - * combination that would make the field say otherwise. + * run is its own root, a child takes its parent's root, and the public API — `root()` and + * `childOf()`, the only public mints, with `copy()` private alongside the constructor — has no way + * to build one that says otherwise. (Reflection still can; see the test below that demonstrates it.) */ class ExtractionRunLineageTest { @@ -93,28 +100,14 @@ class ExtractionRunLineageTest { assertThat(replacementWithNoParent.rootRunRef).isEqualTo(runD) } - @Test - fun `a parentless run whose root is some other run is rejected`() { - assertThatIllegalArgumentException().isThrownBy { - ExtractionRunLineage(runRef = runA, rootRunRef = runB) - }.withMessageContaining("no parent is its own root") - } - - @Test - fun `a run with a parent cannot claim to be its own root`() { - assertThatIllegalArgumentException().isThrownBy { - ExtractionRunLineage(runRef = runA, rootRunRef = runA, parentRunRef = runB) - }.withMessageContaining("takes its parent's root") - } - @Test fun `a run cannot be its own parent or supersede itself`() { assertThatIllegalArgumentException().isThrownBy { - ExtractionRunLineage(runRef = runA, rootRunRef = runB, parentRunRef = runA) + ExtractionRunLineage.childOf(runRef = runA, parent = ExtractionRunLineage.root(runA)) }.withMessageContaining("its own parent") assertThatIllegalArgumentException().isThrownBy { - ExtractionRunLineage(runRef = runA, rootRunRef = runA, supersedesRunRef = runA) + ExtractionRunLineage.root(runRef = runA, supersedesRunRef = runA) }.withMessageContaining("supersede itself") } @@ -126,14 +119,70 @@ class ExtractionRunLineageTest { } @Test - fun `copying a lineage still has to satisfy the root invariant`() { - val child = ExtractionRunLineage.childOf(runB, ExtractionRunLineage.root(runA)) + fun `a root that contradicts the parent chain is unrepresentable through the public API`() { + // PR #95 review comment: any non-self root used to pass the constructor, whether or not it + // named the actual parent's root, and nothing verified it. root() and childOf() are now the + // only way to mint a lineage; childOf() derives rootRunRef from the actual parent lineage + // it is handed, and root() sets it to the run's own ref, so there is no parameter through + // which a caller could hand in an impostor root. + val ctor = ExtractionRunLineage::class.primaryConstructor + assertThat(ctor).isNotNull() + assertThat(ctor!!.visibility).isEqualTo(KVisibility.PRIVATE) + + // copy() follows the constructor's visibility (@ConsistentCopyVisibility on the class), so + // the same closure applies to a caller who already holds a lineage and tries to overwrite + // just the root. + val copyFunction = ExtractionRunLineage::class.declaredFunctions.single { it.name == "copy" } + assertThat(copyFunction.visibility).isEqualTo(KVisibility.PRIVATE) + } + + @Test + fun `reflection can still construct a lineage whose root contradicts its parent`() { + // The fix closes the public API, root() and childOf(). The JVM still lets reflection call + // a private constructor directly, the same route a permissive deserializer would take. This + // proves only that: the call below succeeds and the value it hands back names a root that + // runA's own lineage disagrees with. + val ctor = ExtractionRunLineage::class.primaryConstructor!! + ctor.isAccessible = true + + val impostor = ctor.call(runC, runD, runA, null, 1) + + assertThat(impostor.parentRunRef).isEqualTo(runA) + assertThat(impostor.rootRunRef).isEqualTo(runD) + // runA's own lineage says its root is runA, not runD — the impostor is representable + // anyway, because this constructor call never saw runA's lineage, only its ref. + assertThat(ExtractionRunLineage.root(runA).rootRunRef).isNotEqualTo(impostor.rootRunRef) + } + + @Test + fun `a parentless run whose root names another run is still rejected, reached the one way left`() { + // root()/childOf() can never reach this state — root() always sets rootRunRef to its own + // ref — so the only way left to exercise the "no parent is its own root" init guard is + // the same reflective call a permissive deserializer would make. + val ctor = ExtractionRunLineage::class.primaryConstructor!! + ctor.isAccessible = true + + // Kotlin reflection routes this through java.lang.reflect.Constructor, so the init + // block's exception arrives wrapped in an InvocationTargetException rather than bare. + val thrown = catchThrowable { ctor.call(runA, runB, null, null, 0) } + + assertThat(thrown).isInstanceOf(InvocationTargetException::class.java) + assertThat(thrown.cause) + .isInstanceOf(IllegalArgumentException::class.java) + .hasMessageContaining("no parent is its own root") + } + + @Test + fun `a run cannot claim to be the root of a lineage it is already a member of`() { + // Unlike the parentless case above, this guard ("a run with a parent takes its parent's + // root") is reachable through the public API: childOf() derives rootRunRef from the parent + // it's given, but nothing stops a caller from reusing an ancestor's own ref as the "new" run. + val root = ExtractionRunLineage.root(runA) + val child = ExtractionRunLineage.childOf(runB, root) - // A data class copy runs the same init block, so nothing can drop the parent and keep a - // root that no longer follows from it. assertThatIllegalArgumentException().isThrownBy { - child.copy(parentRunRef = null) - }.withMessageContaining("no parent is its own root") + ExtractionRunLineage.childOf(runRef = runA, parent = child) + }.withMessageContaining("takes its parent's root") } @Test diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/IncrementalPropositionExtractionTest.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/IncrementalPropositionExtractionTest.kt index f5522934..607bcb4b 100644 --- a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/IncrementalPropositionExtractionTest.kt +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/IncrementalPropositionExtractionTest.kt @@ -24,7 +24,6 @@ import com.embabel.agent.core.DataDictionary import com.embabel.agent.rag.model.NamedEntity import com.embabel.agent.rag.service.NamedEntityDataRepository import com.embabel.chat.Message -import com.embabel.dice.common.ConversationAnalysisRequestEvent import com.embabel.dice.common.EntityResolver import com.embabel.dice.common.Relations import com.embabel.dice.common.SourceAnalysisContext @@ -108,10 +107,21 @@ class IncrementalPropositionExtractionTest { // Two entry points, and no third and fourth name to keep in step with them. Everything a // caller supplies beyond the arguments these methods always took rides on the request, so - // a locator, a revision and a profile add no method names and no arities. + // a locator, a revision, a profile and a run reference add no method names and no arities. assertEquals(emptySet>>(), declaredParameterLists("rememberTextFromSource")) assertEquals(emptySet>>(), declaredParameterLists("rememberFileFromSource")) + // The descriptor sets above are the whole published surface, and the run reference is a + // field on the request, so no descriptor mentions it. This is the assertion that says the + // slice adding the run changed no signature: if it had, one of these lists would name the + // type. + (rememberTextParameters + rememberFileParameters).forEach { parameters -> + assertFalse( + ExtractionRunRef::class.java in parameters, + "a run reference must travel on the request, never as a parameter: $parameters", + ) + } + // A request-carrying call can never collapse onto one written without a request: the // request always arrives on the end, at an arity the other caller never fills. (rememberTextParameters + rememberFileParameters).forEach { parameters -> @@ -127,13 +137,16 @@ class IncrementalPropositionExtractionTest { @Test fun `no field, parameter or return type outside the request carries what the request carries`() { - // The point of the request object: a locator, a revision and a profile reach extraction - // through it and through nothing else. A future dimension added as a loose parameter on an - // entry point would put the surface back where it started, so this sweeps the whole class. + // The point of the request object: a locator, a revision, a profile and a run reference + // reach extraction through it and through nothing else. A future dimension added as a + // loose parameter on an entry point would put the surface back where it started, so this + // sweeps the whole class. The run is here because it is the first dimension to arrive + // after the request existed, and it arrived without touching a single signature. val carried = setOf( SourceLocator::class.java, SourceRevisionRef::class.java, ExtractionContentProfileRef::class.java, + ExtractionRunRef::class.java, ) IncrementalPropositionExtraction::class.java.declaredMethods .filter { it.name.startsWith("remember") && '$' !in it.name } @@ -148,16 +161,16 @@ class IncrementalPropositionExtractionTest { } @Test - fun `no entry point, context, constructor, or field anywhere carries a run reference`() { - // PR #94 review comment: buildContext accepted a currentRun and put it on the context. - // persistAndProject — the method that actually saves the extracted propositions — takes - // only a ChunkPropositionResult and never the context that would have carried it. No - // consuming write exists on this branch (the durable run store is DICE #67/#98/#99), so - // the fix removes the no-op parameter. + fun `the run reference reaches the context and nothing consumes it`() { + // PR #94's review comment was that buildContext accepted a currentRun and put it on the + // context while nothing read it, so the parameter came out and the type came out with it. + // The run is back now, on the request, and the same question applies: what reads it? + // Nothing does, and this pins the two facts that make that checkable. // - // persistAndProject has exactly one overload, and it is the one-argument shape: a second - // overload taking the context (a run reference's only possible route back in) would slip - // past a check that only confirms one particular arity exists. + // First: persistAndProject — the method that actually saves the extracted propositions — + // takes only a ChunkPropositionResult and never the context that would carry a run. It has + // exactly one overload, so a second one taking the context (a run reference's only route + // into a write) cannot hide behind a check that only confirms one particular arity exists. val persistAndProjectOverloads = IncrementalPropositionExtraction::class.java .declaredMethods.filter { it.name == "persistAndProject" } assertEquals(1, persistAndProjectOverloads.size, "persistAndProject must have exactly one overload") @@ -167,64 +180,29 @@ class IncrementalPropositionExtractionTest { persistAndProjectOverloads.single().parameterTypes.single(), ) - // The type itself is gone from the classpath, so there is nothing left for a caller to - // depend on — not even a reference to it, let alone a call to a member of it. - assertThrows(ClassNotFoundException::class.java) { - Class.forName("com.embabel.dice.proposition.extraction.ExtractionRunRef") - } + // Second: the run reaches the context by exactly one route, the request, and reaches it + // whole. A run named on a request is the run the pipeline sees, and nothing in between + // rewrites, defaults or drops it. + val pipeline = pipelineReturningNoResult() + val extraction = extraction(pipeline) + val run = ExtractionRunRef("run-1") - // A run reference under another name (runRef, runId, AnalysisRunRef, ...) would satisfy a - // sweep that only recognizes "currentRun"/"extractionRun" literally, so this checks every - // declared member's name for "run" as a substring, not a fixed set of spellings. Verified - // before writing this: none of the five carriers has a legitimate declared method, field, - // or constructor parameter whose name contains "run" today, confirmed by a throwaway - // reflection dump run against the built classes, so this sweep starts from a clean - // baseline and any future match is either a reintroduced run reference or something that - // needs an explicit, named exclusion (there are none right now). - // - // Types are checked via the *generic* signature (genericType / genericReturnType / - // genericParameterTypes), not the erased Class. An erased check sees `List` for a field - // declared `List` and would miss it; `Type.toString()` on a generic - // signature includes the type argument, so the same substring match catches a run - // reference hidden inside a collection or other generic wrapper. Confirmed empirically, - // same as the name sweep: zero matches on the current classes. - // - // Constructor parameters are checked by type only, not name. The JVM does not preserve - // real parameter names in these classes' compiled constructors (reflection reports them - // as arg0, arg1, ...), so a name-based check on a constructor parameter would silently - // never fire; claiming otherwise here would be the same overclaim this test exists to - // avoid making about other code. - // ExtractionRequest is swept alongside the rest: it is where a run reference would land - // once the store that consumes it arrives, so it is the likeliest place for one to - // reappear early. - val carriers = listOf( - IncrementalPropositionExtraction::class.java, - ExtractionRequest::class.java, - SourceAnalysisContext::class.java, - SourceAnalysisRequestEvent::class.java, - ConversationAnalysisRequestEvent::class.java, + extraction.rememberText( + text = "run text", + sourceId = "run:only", + user = user(), + additionalGrounding = emptyList(), + perspective = null, + mintNewEntities = null, + request = ExtractionRequest(currentRun = run), ) - val runInName = Regex("(?i)run") - fun suspectType(type: java.lang.reflect.Type) = runInName.containsMatchIn(type.toString()) - carriers.forEach { type -> - type.declaredMethods.forEach { method -> - assertFalse(runInName.containsMatchIn(method.name), "${type.simpleName}.${method.name}: name mentions run") - assertFalse( - method.genericParameterTypes.any(::suspectType) || suspectType(method.genericReturnType), - "${type.simpleName}.${method.name}: a parameter or return type mentions run", - ) - } - type.declaredFields.forEach { field -> - assertFalse(runInName.containsMatchIn(field.name), "${type.simpleName}.${field.name}: name mentions run") - assertFalse(suspectType(field.genericType), "${type.simpleName}.${field.name}: field type mentions run") - } - type.declaredConstructors.forEach { constructor -> - assertFalse( - constructor.genericParameterTypes.any(::suspectType), - "${type.simpleName} constructor $constructor: a parameter type mentions run", - ) - } - } + + val context = capturedContext(pipeline, "run text", "run:only", emptyList()) + assertSame(run, context.currentRun) + // A run needs no source and no profile of its own: the dimensions stay independent. + assertNull(context.sourceLocator) + assertNull(context.sourceRevision) + assertNull(context.profile) } @Test @@ -463,7 +441,7 @@ class IncrementalPropositionExtractionTest { mintNewEntities: Boolean?, request: ExtractionRequest, ) { - seen += "request:$sourceId:${request.profile?.name}" + seen += "request:$sourceId:${request.profile?.name}:${request.currentRun?.runId}" } } @@ -473,8 +451,22 @@ class IncrementalPropositionExtractionTest { user(), ExtractionRequest(profile = ExtractionContentProfileRef("house-style", "v1")), ) + // A run on its own is something to carry too, so a request holding nothing else still + // routes here and arrives whole. + extraction.rememberFile( + ByteArrayInputStream("legacy file text".toByteArray()), + "run-only.txt", + user(), + ExtractionRequest(currentRun = ExtractionRunRef("run-1")), + ) - assertEquals(listOf("request:remember:profiled.txt:house-style"), seen) + assertEquals( + listOf( + "request:remember:profiled.txt:house-style:null", + "request:remember:run-only.txt:null:run-1", + ), + seen, + ) verifyNoInteractions(pipeline) } @@ -486,6 +478,7 @@ class IncrementalPropositionExtractionTest { val locator = UriLocator("https://example.com/source") val revision = SourceRevisionRef(locator.key(), "r7") val profile = ExtractionContentProfileRef("house-style", "v1") + val run = ExtractionRunRef("run-1") val perspective = mock() val grounding = listOf("record:one", "record:two") @@ -500,13 +493,16 @@ class IncrementalPropositionExtractionTest { sourceLocator = locator, sourceRevision = revision, profile = profile, + currentRun = run, ), ) + // Every field a request can hold, on one call, arriving on the context unchanged. val context = capturedContext(pipeline, "source text", "caller:source:r7", grounding) assertSame(locator, context.sourceLocator) assertSame(revision, context.sourceRevision) assertSame(profile, context.profile) + assertSame(run, context.currentRun) assertSame(perspective, context.perspective) assertEquals(true, context.mintNewEntities) } @@ -838,13 +834,14 @@ class IncrementalPropositionExtractionTest { } @Test - fun `a request carries profile and revision to the context with and without a source`() { + fun `a request carries profile, run and revision to the context with and without a source`() { val pipeline = pipelineReturningNoResult() val extraction = extraction(pipeline) val user = user() val locator = UriLocator("https://example.com/profiled") val revision = SourceRevisionRef(locator.key(), "r1") val profile = ExtractionContentProfileRef("house-style", "v1") + val run = ExtractionRunRef("run-1") extraction.rememberText( text = "untyped text", @@ -853,7 +850,7 @@ class IncrementalPropositionExtractionTest { additionalGrounding = emptyList(), perspective = null, mintNewEntities = null, - request = ExtractionRequest(profile = profile), + request = ExtractionRequest(profile = profile, currentRun = run), ) extraction.rememberText( text = "source text", @@ -866,17 +863,21 @@ class IncrementalPropositionExtractionTest { sourceLocator = locator, sourceRevision = revision, profile = profile, + currentRun = run, ), ) - // A profile needs no source of its own: the two dimensions stay independent on the way in. + // A profile and a run need no source of their own: the dimensions stay independent on the + // way in. val fromUntyped = capturedContext(pipeline, "untyped text", "untyped:profiled", emptyList()) assertSame(profile, fromUntyped.profile) + assertSame(run, fromUntyped.currentRun) assertNull(fromUntyped.sourceLocator) assertNull(fromUntyped.sourceRevision) val fromSource = capturedContext(pipeline, "source text", "source:profiled", emptyList()) assertSame(profile, fromSource.profile) + assertSame(run, fromSource.currentRun) assertSame(locator, fromSource.sourceLocator) assertSame(revision, fromSource.sourceRevision) } @@ -889,12 +890,13 @@ class IncrementalPropositionExtractionTest { val locator = UriLocator("file:///notes/profiled.txt") val revision = SourceRevisionRef(locator.key(), "r1") val profile = ExtractionContentProfileRef("house-style", "v1") + val run = ExtractionRunRef("run-1") extraction.rememberFile( inputStream = ByteArrayInputStream("plain file text".toByteArray()), filename = "profile-only.txt", user = user, - request = ExtractionRequest(profile = profile), + request = ExtractionRequest(profile = profile, currentRun = run), ) extraction.rememberFile( inputStream = ByteArrayInputStream("source file text".toByteArray()), @@ -904,6 +906,7 @@ class IncrementalPropositionExtractionTest { sourceLocator = locator, sourceRevision = revision, profile = profile, + currentRun = run, ), ) // Both file calls hand their text to the request-taking entry point, so the two contexts @@ -920,6 +923,7 @@ class IncrementalPropositionExtractionTest { ) contextCaptor.allValues.forEach { context -> assertSame(profile, context.profile) + assertSame(run, context.currentRun) } val fromSource = contextCaptor.allValues.single { it.sourceLocator != null } assertSame(locator, fromSource.sourceLocator) @@ -927,7 +931,7 @@ class IncrementalPropositionExtractionTest { } @Test - fun `calls written without a request carry no profile`() { + fun `calls written without a request carry no profile and no run`() { val pipeline = pipelineReturningNoResult() val extraction = extraction(pipeline) val user = user() @@ -950,16 +954,18 @@ class IncrementalPropositionExtractionTest { ) contextCaptor.allValues.forEach { context -> assertNull(context.profile) + assertNull(context.currentRun) } } @Test - fun `a profile changes nothing else about what the pipeline is asked to do`() { + fun `a profile and a run change nothing else about what the pipeline is asked to do`() { val pipeline = pipelineReturningNoResult() val extraction = extraction(pipeline) val user = user() val grounding = listOf("record:one") val profile = ExtractionContentProfileRef("house-style", "v1") + val run = ExtractionRunRef("run-1") extraction.rememberText( text = "same text", @@ -976,7 +982,7 @@ class IncrementalPropositionExtractionTest { additionalGrounding = grounding, perspective = ExtractionPerspective.USER, mintNewEntities = true, - request = ExtractionRequest(profile = profile), + request = ExtractionRequest(profile = profile, currentRun = run), ) val contextCaptor = argumentCaptor() @@ -990,40 +996,48 @@ class IncrementalPropositionExtractionTest { ) val (plain, profiled) = contextCaptor.allValues - // Comparing whole contexts is the point: they agree on every component but the one the + // Comparing whole contexts is the point: they agree on every component but the two the // second call set. The resolver is substituted because buildContext constructs a fresh // one per call by design, so it is never the same instance twice. assertEquals( plain, - profiled.copy(profile = null, entityResolver = plain.entityResolver), + profiled.copy(profile = null, currentRun = null, entityResolver = plain.entityResolver), ) assertSame(profile, profiled.profile) + assertSame(run, profiled.currentRun) } @Test - fun `event profile reaches the context observed by the pipeline`() { + fun `event profile and run reach the context observed by the pipeline`() { val pipeline = pipelineReturningNoResult() val extraction = extraction(pipeline) val source = mock>() whenever(source.id).thenReturn("event-source") whenever(source.size).thenReturn(1) val profile = ExtractionContentProfileRef("house-style", "v1") + val run = ExtractionRunRef("run-1") val profileCalls = AtomicInteger() + val runCalls = AtomicInteger() val event = object : SourceAnalysisRequestEvent(this, user()) { override fun incrementalSource(): IncrementalSource = source override fun profile(): ExtractionContentProfileRef = profile.also { profileCalls.incrementAndGet() } + + override fun currentRun(): ExtractionRunRef = + run.also { runCalls.incrementAndGet() } } extraction.extractPropositions(event) - // One read: the async path builds one context through the same buildContext the direct - // calls use, so there is nowhere else for a second read to happen. + // One read each: the async path builds one context through the same buildContext the + // direct calls use, so there is nowhere else for a second read to happen. assertEquals(1, profileCalls.get()) + assertEquals(1, runCalls.get()) val contextCaptor = argumentCaptor() verify(pipeline).processChunk(any(), contextCaptor.capture()) assertSame(profile, contextCaptor.firstValue.profile) + assertSame(run, contextCaptor.firstValue.currentRun) assertNull(contextCaptor.firstValue.sourceLocator) } diff --git a/dice/src/test/kotlin/com/embabel/dice/provenance/SourceRevisionContractTest.kt b/dice/src/test/kotlin/com/embabel/dice/provenance/SourceRevisionContractTest.kt index 31ef0355..bdf51688 100644 --- a/dice/src/test/kotlin/com/embabel/dice/provenance/SourceRevisionContractTest.kt +++ b/dice/src/test/kotlin/com/embabel/dice/provenance/SourceRevisionContractTest.kt @@ -47,6 +47,37 @@ class SourceRevisionContractTest { .isThrownBy { SourceRevisionRef(locator.key(), "\t") } } + @Test + fun `source revision ref accepts any value up to the bounds it owns, and refuses past them`() { + // docs/design/source-revisions.md: a revision is opaque to DICE — compared for exact + // equality, never parsed. The one limit on either half is SourceIdentityBounds, checked + // here, on the type that owns the value. Both ends of that boundary are pinned, because + // everything downstream — a revision query, run recording — accepts whatever this + // constructor accepts and adds no cap of its own, so this is the only place length can + // decide anything. + val atCeiling = SourceRevisionRef( + "u".repeat(SourceIdentityBounds.MAX_SOURCE_KEY_LENGTH), + "r".repeat(SourceIdentityBounds.MAX_SOURCE_REVISION_LENGTH), + ) + + assertThat(atCeiling.sourceKey).hasSize(SourceIdentityBounds.MAX_SOURCE_KEY_LENGTH) + assertThat(atCeiling.sourceRevision).hasSize(SourceIdentityBounds.MAX_SOURCE_REVISION_LENGTH) + + assertThatIllegalArgumentException() + .isThrownBy { + SourceRevisionRef("u".repeat(SourceIdentityBounds.MAX_SOURCE_KEY_LENGTH + 1), "r1") + } + .withMessageContaining("source key") + assertThatIllegalArgumentException() + .isThrownBy { + SourceRevisionRef( + locator.key(), + "r".repeat(SourceIdentityBounds.MAX_SOURCE_REVISION_LENGTH + 1), + ) + } + .withMessageContaining("source revision") + } + @Test fun `provenance revision participates in equality and deduplication`() { val revisionless = ProvenanceEntry(locator = locator) diff --git a/docs/design/extraction-runs.md b/docs/design/extraction-runs.md index c69114b9..c2183654 100644 --- a/docs/design/extraction-runs.md +++ b/docs/design/extraction-runs.md @@ -117,8 +117,7 @@ The root is redundant with the parent chain, and it is stored anyway. OpenLineag consumers do not have to walk the chain a hop at a time. Deep pass-and-retry chains are exactly where walking hurts, and the audit projection reads lineage by run. -A denormalized field is only worth having if it cannot drift, so it is fixed at mint and the -constructor rejects every combination that would make it a lie: +A denormalized field is only worth having if it cannot drift, so it is fixed at mint: - a run with no parent is its own root; - a run with a parent takes its parent's root, and therefore is not its own root; @@ -127,6 +126,23 @@ constructor rejects every combination that would make it a lie: `ExtractionRunLineage.root(...)` and `.childOf(...)` do the arithmetic, and `childOf` defaults the pass index to the parent's plus one. +**The constructor no longer takes a root a caller could get wrong.** A PR #95 review comment +pointed out that the old constructor took `rootRunRef` as an independent parameter: any non-self +value passed, whether or not it named the actual parent's root, and nothing checked it against the +parent. The constructor is now private and `copy()` follows it — `@ConsistentCopyVisibility` on +the class — so `root()` and `childOf()` are the only way in: `root()` sets the root to the run's +own ref, `childOf()` derives it from the actual parent lineage it is handed. + +That closes the public API and stops there. Kotlin reflection can still call the private +constructor directly and hand it a root that contradicts the parent it names — the same route a +Jackson deserializer resolving a data class's primary constructor would take. Nothing serializes an +`ExtractionRunLineage` today, so this is a residual for whoever builds that wiring next. It is +written down here so it stays known: a future store slice reconstructing a lineage from stored +fields has to walk through `childOf()` with the parent's own lineage in hand; re-assembling +`rootRunRef` and `parentRunRef` from separate columns is the shortcut that closes. `ExtractionRunLineageTest` +pins both halves — that the public surface is closed, and that the reflective call still succeeds +and produces an inconsistent root. + What a value type cannot check is a cycle of length two or more: that needs the other runs, so bounded cycle-safe traversal belongs to the store that walks the chains. @@ -231,7 +247,7 @@ of the ones that matter. That is a better position than having adopted a naming ## The cap rule -Every string a run stores is bounded, the bound is a named constant on `ExtractionRunLimits`, the +Most strings a run stores are bounded: the bound is a named constant on `ExtractionRunLimits`, the check runs in the `init` block of the type that owns the value, and anything over the bound is **rejected**. Truncating an identifier would be worse than rejecting it: a shortened id is a different id, and a store would then key rows on a value the caller never minted. @@ -239,7 +255,6 @@ different id, and a store would then key rows on a value the caller never minted | Constant | Value | Applies to | | --- | --- | --- | | `MAX_IDENTIFIER_LENGTH` | 256 | opaque tokens, fingerprints, model and role names, service names, provider response ids, runtime identifiers | -| `MAX_SOURCE_KEY_LENGTH` | 1024 | source keys, which come out of `SourceLocator.key()` and can legitimately hold a long URL | | `MAX_FAILURE_DETAIL_LENGTH` | 512 | the one free-text field | | `MAX_SOURCE_REVISIONS` | 256 | source revisions per run | | `MAX_INVOCATIONS` | 1024 | invocation records per run, across every call and attempt | @@ -252,11 +267,14 @@ constructor still rejects a longer one. Lengths count UTF-16 chars, so a 256-char identifier can be around 1 KB of UTF-8. The bound exists to keep a run header finite. -`SourceRevisionRef` predates this rule and validates non-blank only, so `ExtractionRun` applies the -source-key and identifier bounds to the revisions it stores. That is a stopgap: the check belongs -on `SourceRevisionRef` itself, so the bound travels with the type instead of being re-applied by -every consumer. Moving it is a one-line change in the module that owns that type and is follow-up -work. +`sourceKey` and `sourceRevision` are not part of the cap rule and carry no bound on `ExtractionRun`. +A revision's contract is defined where the type lives — +[docs/design/source-revisions.md](source-revisions.md): opaque, compared for exact equality, never +parsed, and bounded once by `SourceIdentityBounds` in `SourceRevisionRef`'s own constructor. A PR +#95 review comment caught `ExtractionRun` adding a second length cap on top of that contract, so a +revision an earlier query accepted could still fail run recording; the fix deletes the run's cap and +leaves the one on the type that owns the value. A run records whatever a `SourceRevisionRef` can +hold, which is the property that makes "accepted by a query, therefore recordable" true. One string sits outside the rule and stays outside it: `ContextId.value`, the tenant, which is validated non-blank and not bounded. `ContextId` is a DICE-wide type owned by the agent framework, From 7030e69decc410f507e58ae7434f829e52b954df Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Wed, 2 Sep 2026 00:51:49 -0400 Subject: [PATCH 3/7] Close the failure-detail leak with a typed vocabulary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ExtractionFailure carried free text into durable storage: truncation and line-break flattening bounded its size while prompt content, PII and credentials rode through untouched. The type now holds a code, an optional stage, a provider status in the HTTP range, and a unit-named measure paired with a number — no String or Throwable parameter exists anywhere in its reachable shape, so raw source text cannot be represented. Reflection canaries pin the absence of any free-text route. ProtectedContentRef returns as specification only: the host owns writer, reader, retention and expiry, and DICE stores the handle's contract, never its content. The envelope's metamodelFingerprint names the extraction coordinator as its producer, and per-proposition schema attribution answers through run lineage. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com> --- CHANGELOG.md | 54 ++++- .../extraction/ExtractionFailure.kt | 226 ++++++++++++------ .../proposition/extraction/ExtractionRun.kt | 13 +- .../extraction/ExtractionRunEnvelope.kt | 16 +- .../extraction/ExtractionRunLimits.kt | 32 ++- .../extraction/ProtectedContentRef.kt | 91 +++++++ .../ExtractionFailureVocabularyTest.kt | 212 ++++++++++++++++ .../extraction/ExtractionRunContractTest.kt | 10 +- .../extraction/ExtractionRunFixtures.kt | 17 +- .../extraction/ExtractionRunPrivacyTest.kt | 97 ++++---- .../extraction/ExtractionRunValueTypesTest.kt | 19 +- docs/design/extraction-runs.md | 96 ++++++-- 12 files changed, 690 insertions(+), 193 deletions(-) create mode 100644 dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ProtectedContentRef.kt create mode 100644 dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionFailureVocabularyTest.kt diff --git a/CHANGELOG.md b/CHANGELOG.md index 0e5d0aed..93894060 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1199,9 +1199,13 @@ and the consumer PRs that deliver it). - **EXPERIMENTAL.** The extraction run model in `dice` core — the value types DICE #67's store, lineage and wiring slices build on. `ExtractionRun`, keyed by (`ContextId`, `ExtractionRunRef`) through the new `ExtractionRunKey`, records the profile version in force, the ordered source - revisions it read, its lineage, prompt/schema/metamodel fingerprints, extractor/host/runtime + revisions it read, its lineage, prompt/schema/metamodel fingerprints (the metamodel one is written + by the extraction coordinator in a later slice, which resolves the declared schema stamp from the + host's `DeclaredSchemaSource` and hashes it; per-proposition schema attribution is answered by + following a proposition back to its run through run lineage), extractor/host/runtime identity, requested model configuration, pseudonymous subject references, experiment and cohort - labels, status, timing, counts, invocation records, bounded sanitized failures, and an explicit + labels, status, timing, counts, invocation records, bounded failures in a closed vocabulary, and + an explicit replay-fidelity value. Nothing stores one yet: the lifecycle state machine and the store contract are the next slice, and no code constructs a run during extraction until the wiring slice. **Requested and observed model facts are separate types, structurally.** @@ -1246,12 +1250,37 @@ and the consumer PRs that deliver it). email address, a URL, a file path, a JSON fragment and a human name outright. The KDoc states what that does not prove — a value type cannot tell a pseudonym from a username — rather than implying a guarantee. A token's `toString` shows eight characters and a validation message never quotes the - value it rejected. `ExtractionFailure` is a classified `ExtractionFailureCode` plus a bounded - single-line detail; `fromThrowable`, the path DICE itself uses, never reads `Throwable.message` - and records exception class names down a bounded cycle-safe cause chain, because a provider quotes - the prompt back in its message. Tests extract from a fixture whose source text is known and assert - no fragment of it, no address shape, no link shape and no long digit run survives into a - field-by-field dump of a fully populated run. **Replay fidelity never claims exact replay.** + value it rejected. **Free text cannot reach durable storage through `ExtractionFailure`.** The + record speaks a closed vocabulary and has no `String` parameter, property or field, and no factory + taking a `Throwable`: a classified `ExtractionFailureCode` (11 values), an optional + `ExtractionFailureStage` (10 values) saying where in the run's work it happened, an optional + provider status bounded to 100..599, and an optional `ExtractionFailureMeasure` pairing one number + with an `ExtractionFailureQuantity` that names its unit — `TOKEN_COUNT`, `ELAPSED_MILLIS`, + `RETRY_AFTER_SECONDS` — so 4096 can never be recorded without saying it is tokens. The earlier + shape, a code plus a bounded whitespace-flattened `detail` string, closed nothing a #95 review + comment cared about: a truncated prompt is still a prompt, and a credential, an email address or a + paragraph of protected text all fit in 512 characters. The vocabulary closes it at construction, + which is why `detail`, `of(code, detail, ...)`, `fromThrowable` and `MAX_FAILURE_DETAIL_LENGTH` are + all gone. "Chunk 3 of 12 exceeded the token budget" survives as `invocation.invocationIndex` plus a + `TOKEN_COUNT` measure; the sentence, the part nobody could vouch for, does not. A host that wants + the exception message keeps it under its own retention and access rules. Canary tests take raw + source text, a prompt fragment, an email address and two credential shapes a scanner recognises, + and assert no constructor, factory or method will take any of them, that no type a failure reaches + has a text field, and that every value a fully populated failure holds is an enum, a number or an + instant. Tests still extract from a fixture whose source text is known and assert no fragment of + it, no address shape, no link shape and no long digit run survives into a field-by-field dump of a + fully populated run. + **`ProtectedContentRef` ships as specification only.** One interface, two members — an opaque + `handle` and an `expiresAt` — and no implementation, no production reference, and nothing DICE + stores. It is the written contract for a host keeping detailed failure material of its own, and it + hands the host all three jobs: the writer, because DICE never sees the material; the reader, + because resolving a handle runs under the host's access rules and the handle grants no access on + its own; and retention, where `expiresAt` is the host's declaration and the host's own job is what + makes it true. The KDoc carries a worked example of a host writing an exception message into its + vault under a handle for ninety days. A type of this name shipped in an earlier #98 draft as a + stored value and was deleted during review, because nothing attached it to a run and DICE had no + writer, reader or retention behaviour behind it; a test now asserts the interface is abstract and + that no compiled DICE class mentions the type. **Replay fidelity never claims exact replay.** `ExtractionReplayFidelity` is `NONE`, `METADATA`, `APPROXIMATE`; the strongest value is still approximate and `strongest()` returns it so appending a value cannot quietly strengthen the claim. **`ExtractionRunStatus` ships the four values only** — `RUNNING`, `COMPLETED`, `FAILED`, @@ -1266,10 +1295,9 @@ and the consumer PRs that deliver it). and a migration later. **One cap rule**: every bound is a named constant on `ExtractionRunLimits`, checked in the `init` block of the type that owns the value, and an over-long value is rejected rather than truncated, because a shortened identifier is a different identifier. Identifiers cap - at 256 characters, the one free-text failure detail at 512, and the three collections at 256 - source revisions, 1024 invocation records and 64 failures. The failure detail is the single - exception to rejection and only on the way in: its factories clip it, the constructor still - rejects. `sourceKey` and `sourceRevision` carry no length bound on `ExtractionRun`: their one + at 256 characters, a provider status falls in 100..599, and the three collections cap at 256 + source revisions, 1024 invocation records and 64 failures. The rule now has no exception, because + the model has no free-text field for one to apply to. `sourceKey` and `sourceRevision` carry no length bound on `ExtractionRun`: their one bound lives on `SourceRevisionRef`, the type that owns those strings, which checks both halves against `SourceIdentityBounds` at construction (`docs/design/source-revisions.md`). A run accepts whatever that type accepts and adds no cap of its own, so a revision a query worked with can @@ -1288,6 +1316,8 @@ and the consumer PRs that deliver it). **Compatibility: additive, new types only.** Nothing existing changes. No existing class gains or loses a member, no signature moves, no default changes, and no behaviour differs — this slice adds types to `com.embabel.dice.proposition.extraction` and touches nothing that was already there. + `ExtractionFailure`'s reshape is inside this slice: the type is new here, so nothing released has + ever seen the `detail` field. Source, binary and Java compatibility are therefore all unaffected, and the scoped Kotlin ABI boundary the Wave A and B slices declared does not apply because no existing data class gained a field. No stored data changes and no migration is required: nothing serializes a run yet, and the diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionFailure.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionFailure.kt index 09f51ce3..a8ee7a5d 100644 --- a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionFailure.kt +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionFailure.kt @@ -21,9 +21,9 @@ import java.time.Instant /** * What kind of thing went wrong, from a fixed list. * - * The code is what a query groups by and what an alert fires on, so it is an enum rather than a - * string a caller invents per site. Anything that does not fit is [INTERNAL] or [UNCLASSIFIED], - * and a code that keeps getting used for the wrong thing is a signal to add one. + * The code is what a query groups by and what an alert fires on, so it comes from a fixed list + * that no caller can extend at a call site. Anything that does not fit is [INTERNAL] or + * [UNCLASSIFIED], and a code that keeps getting used for the wrong thing is a signal to add one. * * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. */ @@ -65,24 +65,150 @@ enum class ExtractionFailureCode { } /** - * One thing that went wrong during a run: a code, a short detail, when, and which invocation. + * Where in a run's work the failure happened, from a fixed list. + * + * [ExtractionFailureCode] says what went wrong and this says where, which are different questions: + * an [ExtractionFailureCode.INTERNAL] during [CHUNKING] and one during [PERSISTENCE] send an + * on-call engineer to two different places. + * + * Nothing cross-checks a stage against a code. Most pairings that look impossible turn out to + * happen — a decode failure while persisting is a real thing when a store reads a value back — and + * a record that refuses to state an awkward truth is worse evidence than one that states it. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + */ +@ApiStatus.Experimental +enum class ExtractionFailureStage { + + /** Laying out the call plan, before any request went out. */ + PLANNING, + + /** Reading the source material at the revision the run was asked for. */ + SOURCE_READ, + + /** Splitting source material into the pieces each call would cover. */ + CHUNKING, + + /** Building the prompt for a call. */ + PROMPT_RENDER, + + /** The call to the model provider, from dispatch to response. */ + MODEL_CALL, + + /** Turning the provider's response into the shape the run expected. */ + RESPONSE_DECODE, + + /** Checking a decoded response against the schema and metamodel in force. */ + SCHEMA_CHECK, + + /** Running extracted propositions past the gates. */ + GATING, + + /** Resolving entity mentions to entities. */ + ENTITY_RESOLUTION, + + /** Storing what the run produced. */ + PERSISTENCE, +} + +/** + * What a number attached to a failure counts, with its unit in the name. + * + * A bare number on a failure record is a unit-mismatch bug waiting to happen: 30000 is half a + * minute or thirty seconds or thirty thousand tokens depending on who wrote it. Every value here + * names both the thing and the unit, so a reader and a dashboard agree without a convention to + * remember. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + */ +@ApiStatus.Experimental +enum class ExtractionFailureQuantity { + + /** Tokens the failure was about: a budget that was exceeded, or what a call asked for. */ + TOKEN_COUNT, + + /** Characters of material the failure was about. */ + CHARACTER_COUNT, + + /** Bytes the failure was about. */ + BYTE_COUNT, + + /** How many things were involved — chunks, propositions, entity mentions. */ + ITEM_COUNT, + + /** Milliseconds that had passed when the work gave up. */ + ELAPSED_MILLIS, + + /** The time limit in force, in milliseconds. */ + LIMIT_MILLIS, + + /** How many seconds the provider asked the caller to wait before trying again. */ + RETRY_AFTER_SECONDS, +} + +/** + * One number a failure record carries, together with what it counts. + * + * Pairing them in a type is the point: there is no way to record 4096 without saying it is tokens, + * and no way to say "tokens" without a number. + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + * + * @property quantity What the number counts, and in what unit + * @property value The number + */ +@ApiStatus.Experimental +data class ExtractionFailureMeasure( + val quantity: ExtractionFailureQuantity, + val value: Long, +) { + + init { + require(value >= 0) { "$quantity must not be negative, was $value" } + } + + override fun toString(): String = "$quantity=$value" + + companion object { + + /** Java-friendly factory. */ + @JvmStatic + fun of(quantity: ExtractionFailureQuantity, value: Long): ExtractionFailureMeasure = + ExtractionFailureMeasure(quantity, value) + } +} + +/** + * One thing that went wrong during a run, said entirely in a closed vocabulary. * * A failure record is evidence, and evidence about a failure is where source text leaks. A model * provider routinely quotes the prompt back in its exception message, a decode error carries the - * fragment it choked on, and both end up in a stored run header if someone writes - * `e.message` into one. So the DICE-minted path, [fromThrowable], never reads - * `Throwable.message` at all: it records the exception class names down the cause chain and - * nothing else, which cannot contain source text because it never touched any. + * fragment it choked on, and both used to reach a stored run header the moment someone wrote + * `e.message` into a text field. So this record has no text field, and there is no constructor, + * factory, or property here that will accept a `String` or a `Throwable`. Prompt content, personal + * data, credentials and protected material have no route into durable storage through this type, + * because there is nothing shaped to hold them. + * + * It holds five things a dashboard can group by and a person can read: the [code], the [stage] it + * happened in, the provider's own [providerStatus], one [measure] answering "how much", and the + * [invocation] it belongs to. That last one already carries the call's ordinal in the plan, so + * "chunk 3 failed" is expressible without a sentence about chunk 3. + * + * One measure per record, deliberately. A failure answers one "how much"; a host that wants the + * exceeded budget *and* the elapsed time is describing two facts, and the second one belongs with + * the rest of its detail. * - * [of] exists for the case where a caller genuinely knows something useful ("chunk 3 of 12 - * exceeded the token budget"). DICE cannot check what a caller puts there. It bounds it, flattens - * it to a single line so a pasted stack trace does not fit, and the contract is that the caller - * supplies a classification rather than a payload. + * **Where the detail goes.** A host that needs the exception message, the response body, or the + * fragment that failed to parse keeps that material itself, under its own retention and access + * rules. `ProtectedContentRef` is the specification for the reference such a host hands around; + * DICE writes nothing behind it and stores none of it. * * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. * * @property code How the failure is classified - * @property detail Short, single-line, sanitized explanation. Empty when the code says it all. + * @property stage Where in the run's work it happened, when that is known + * @property providerStatus The status the provider returned, as an HTTP status code + * @property measure One number about the failure, with its unit * @property at When the failure was recorded * @property invocation The invocation and attempt this failure belongs to, or null for a failure * that happened outside any model call @@ -90,88 +216,36 @@ enum class ExtractionFailureCode { @ApiStatus.Experimental data class ExtractionFailure @JvmOverloads constructor( val code: ExtractionFailureCode, - val detail: String = "", + val stage: ExtractionFailureStage? = null, + val providerStatus: Int? = null, + val measure: ExtractionFailureMeasure? = null, val at: Instant = Instant.now(), val invocation: ExtractionInvocationId? = null, ) { init { - require(detail.length <= ExtractionRunLimits.MAX_FAILURE_DETAIL_LENGTH) { - "detail must be at most ${ExtractionRunLimits.MAX_FAILURE_DETAIL_LENGTH} characters, " + - "was ${detail.length}" - } - require(detail.none { it == '\n' || it == '\r' }) { - "detail must be a single line; a multi-line detail is a stack trace or a quoted payload" - } + requireProviderStatus(providerStatus) } companion object { - /** How many links of a cause chain [fromThrowable] walks. */ - const val MAX_CAUSE_CHAIN: Int = 5 - - /** - * Records a failure with a detail the caller wrote. - * - * Whitespace collapses to single spaces and the result is clipped to - * [ExtractionRunLimits.MAX_FAILURE_DETAIL_LENGTH]. The caller is responsible for the - * detail holding no source text, no prompt, and no personal data; nothing here can verify - * that. [fromThrowable] is the path to use when the detail would have come from an - * exception. - */ + /** Java-friendly factory. */ @JvmStatic @JvmOverloads fun of( code: ExtractionFailureCode, - detail: String, - at: Instant = Instant.now(), - invocation: ExtractionInvocationId? = null, - ): ExtractionFailure = ExtractionFailure( - code = code, - detail = sanitize(detail), - at = at, - invocation = invocation, - ) - - /** - * Records a failure from a throwable, using its class names and nothing else. - * - * The detail is the exception class name, then up to [MAX_CAUSE_CHAIN] causes joined by - * ` <- `. `Throwable.message`, suppressed exceptions, and the stack trace are all - * untouched, so the run header cannot pick up the prompt, the response body, or the - * fragment that failed to parse. - * - * A host that wants the message keeps it in its own logs, where retention and access are - * its to set. - */ - @JvmStatic - @JvmOverloads - fun fromThrowable( - code: ExtractionFailureCode, - throwable: Throwable, + stage: ExtractionFailureStage? = null, + providerStatus: Int? = null, + measure: ExtractionFailureMeasure? = null, at: Instant = Instant.now(), invocation: ExtractionInvocationId? = null, ): ExtractionFailure = ExtractionFailure( code = code, - detail = sanitize(causeChain(throwable).joinToString(" <- ")), + stage = stage, + providerStatus = providerStatus, + measure = measure, at = at, invocation = invocation, ) - - private fun causeChain(throwable: Throwable): List { - val names = mutableListOf() - var current: Throwable? = throwable - val seen = mutableSetOf() - while (current != null && names.size < MAX_CAUSE_CHAIN && seen.add(current)) { - names += current.javaClass.name - current = current.cause - } - return names - } - - private fun sanitize(detail: String): String = - detail.replace(WHITESPACE, " ").trim().take(ExtractionRunLimits.MAX_FAILURE_DETAIL_LENGTH) - - private val WHITESPACE = Regex("\\s+") } } diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRun.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRun.kt index 61ae08df..009bcebe 100644 --- a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRun.kt +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRun.kt @@ -63,8 +63,9 @@ data class ExtractionRunKey( * The second is that the run holds no content. No prompts, no source text, no responses, no * user or session objects, no provider SDK payloads, no extension maps. What a host would have * needed those for is covered by digests it can compare ([fingerprints]) and by bounded - * pseudonymous tokens it can group by ([subjectRefs]). Failures are classified codes with a short - * sanitized detail, and the path DICE itself uses never reads an exception message. + * pseudonymous tokens it can group by ([subjectRefs]). Failures are said in a closed vocabulary of + * codes, stages and numbers, so there is no text field on a run for an exception message to land + * in. * * **What is not decided here.** Which status transitions are legal, which are compare-and-set, and * what a store does with a repeated terminal write belong to the run store contract in the next @@ -101,7 +102,7 @@ data class ExtractionRunKey( * @property replayFidelity How much of this run someone could set up again from what it recorded * @property counts How much the run got through * @property invocations One record per attempt at each planned model call - * @property failures Bounded, sanitized record of what went wrong + * @property failures Bounded record of what went wrong, in the failure vocabulary */ @ApiStatus.Experimental class ExtractionRun @JvmOverloads constructor( @@ -132,7 +133,7 @@ class ExtractionRun @JvmOverloads constructor( val invocations: List = Collections.unmodifiableList(ArrayList(invocations)) - /** What went wrong, bounded and sanitized. */ + /** What went wrong, bounded and said in the failure vocabulary. */ val failures: List = Collections.unmodifiableList(ArrayList(failures)) @@ -259,8 +260,8 @@ class ExtractionRun @JvmOverloads constructor( /** * A summary: identity, lineage, state, and sizes. * - * It leaves out the digests, the reference tokens and the failure details, so a run logged at - * an error site does not spread them. Anything that needs every field reads the properties. + * It leaves out the digests and the reference tokens, so a run logged at an error site does not + * spread them. Anything that needs every field reads the properties. */ override fun toString(): String = "ExtractionRun(contextId=${contextId.value}, runId=${ref.runId}, rootRunId=${rootRef.runId}, " + diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunEnvelope.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunEnvelope.kt index f15099bc..cdb40738 100644 --- a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunEnvelope.kt +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunEnvelope.kt @@ -25,14 +25,24 @@ import org.jetbrains.annotations.ApiStatus * storing the prompt. A host that changes a template and forgets to change its fingerprint gets * runs it cannot tell apart, which is the host's contract to keep. * - * Storing digests instead of the material is deliberate: a prompt template holds instructions and - * often examples, and examples are where real content ends up. + * Storing a digest and leaving the material behind is deliberate: a prompt template holds + * instructions and often examples, and examples are where real content ends up. + * + * **Who writes [metamodelFingerprint].** The extraction coordinator, which arrives in a later + * slice. It reads the declared schema stamp from the host's `DeclaredSchemaSource`, hashes the + * content, and writes the hash here, once per run. Nothing in this slice produces one, so a run + * built today carries whatever its caller passed. + * + * The fingerprint says what the whole run ran under. Asking which schema a single proposition was + * extracted under is answered by following that proposition back to its run through run lineage, + * which is why no per-proposition schema stamp exists. * * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. * * @property promptTemplateFingerprint Digest of the prompt or template the run used * @property schemaFingerprint Digest of the output schema the run asked the model to satisfy - * @property metamodelFingerprint Digest of the metamodel version in force + * @property metamodelFingerprint Content hash of the metamodel version in force, written by the + * extraction coordinator from the host's declared schema source */ @ApiStatus.Experimental data class ExtractionRunFingerprints @JvmOverloads constructor( diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLimits.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLimits.kt index 12ede88d..13438b5d 100644 --- a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLimits.kt +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunLimits.kt @@ -25,10 +25,9 @@ import org.jetbrains.annotations.ApiStatus * construction. Truncating an identifier would be worse than rejecting it — a shortened id is a * different id, and a store would then key rows on a value the caller never minted. * - * A failure detail is the one exception, and only on the way in. It is the model's only free-text - * field, so [ExtractionFailure.of] and [ExtractionFailure.fromThrowable] shorten it to - * [MAX_FAILURE_DETAIL_LENGTH] before construction; keeping a clipped failure record beats losing - * the failure. The constructor still rejects a longer one. + * The rule has no exception, because the model has no free-text field for one to apply to. + * [ExtractionFailure] says everything it says in enums and numbers, so the only bounds it needs + * are the range a provider status can fall in. * * Lengths count UTF-16 chars (`String.length`), so a 256-char identifier can be around 1 KB of * UTF-8. The bound is there to keep a run header finite. @@ -56,11 +55,11 @@ object ExtractionRunLimits { */ const val MAX_IDENTIFIER_LENGTH: Int = 256 - /** - * Longest failure detail a run stores. Long enough for a classified one-line explanation, - * short enough that a run with the full [MAX_FAILURES] of them stays small. - */ - const val MAX_FAILURE_DETAIL_LENGTH: Int = 512 + /** Lowest status code a provider can report on a failure record. */ + const val MIN_PROVIDER_STATUS: Int = 100 + + /** Highest status code a provider can report on a failure record. */ + const val MAX_PROVIDER_STATUS: Int = 599 /** Most source revisions one run may record. */ const val MAX_SOURCE_REVISIONS: Int = 256 @@ -91,6 +90,21 @@ internal fun requireBoundedIdentifier(value: String?, field: String): String? { return value } +/** + * Checks a provider status that may be absent and otherwise has to be a real HTTP status. + * + * The range is what makes the field unable to hold anything but a status: a caller with a number + * that means something else has nowhere to put it here. + */ +internal fun requireProviderStatus(value: Int?): Int? { + if (value == null) return null + require(value in ExtractionRunLimits.MIN_PROVIDER_STATUS..ExtractionRunLimits.MAX_PROVIDER_STATUS) { + "providerStatus must be an HTTP status between ${ExtractionRunLimits.MIN_PROVIDER_STATUS} " + + "and ${ExtractionRunLimits.MAX_PROVIDER_STATUS}, was $value" + } + return value +} + /** Checks a count that may be absent and can never be negative. */ internal fun requireNonNegative(value: Int?, field: String): Int? { if (value == null) return null diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ProtectedContentRef.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ProtectedContentRef.kt new file mode 100644 index 00000000..275c3766 --- /dev/null +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ProtectedContentRef.kt @@ -0,0 +1,91 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import org.jetbrains.annotations.ApiStatus +import java.time.Instant + +/** + * The shape of a reference to failure material a host keeps for itself. Specification only. + * + * [ExtractionFailure] speaks a closed vocabulary of codes, stages and numbers, so an exception + * message, a response body, or the fragment that failed to parse has no route into anything DICE + * stores. Some of that material is still worth keeping, and a host that wants it keeps it in its + * own vault under its own rules. This interface says what such a reference looks like when the + * host passes one around its own code. + * + * **DICE implements none of this and holds none of it.** There is no implementation in this + * repository, nothing in DICE constructs or accepts one, and no run, failure, or stored row has a + * field of this type. It is here so a host writing that vault has a written contract to work from. + * + * ## The three jobs the host owns + * + * **The writer is the host's.** Whatever puts the material in the vault is host code. DICE never + * sees the material, so it can never be the thing that writes it. + * + * **The reader is the host's.** Resolving [handle] back to material is a host operation, subject to + * the host's own access rules. DICE never resolves a handle and never presents one to anything; + * the handle names a row in the host's store, and it is not a way to reach that row. A handle that + * is a signed URL, a pre-authenticated link, a bearer token, a decryption key, or a path on a share + * breaks the contract, because holding a reference has to grant nothing. + * + * **Retention is the host's, and [expiresAt] is where the host writes it down.** The material is + * expected to be gone by that instant. Nothing in DICE sweeps, deletes, or checks it, so the + * declaration is worth exactly what the host's retention job makes it worth. An erasure request + * that has to reach this material reaches it through the host's vault. A reference whose expiry has + * passed stays meaningful as history: a failure that had detail until March and has none now is a + * fact about that failure. + * + * ## A worked example + * + * A host wants the provider's exception message for the ninety days its incident process runs on. + * It mints a handle, writes the message into its own vault under that handle, and keeps the + * reference beside its own incident record. The failure DICE stores carries the code, the stage and + * the invocation, and none of the message. + * + * ```kotlin + * class VaultedFailureDetail( + * override val handle: String, + * override val expiresAt: Instant, + * ) : ProtectedContentRef + * + * // Host code, in the host's own application: + * fun recordDetail(vault: DetailVault, thrown: Throwable): ProtectedContentRef { + * val handle = "pcr:" + UUID.randomUUID() + * val expiresAt = Instant.now().plus(90, ChronoUnit.DAYS) + * vault.put(handle, thrown.stackTraceToString(), expiresAt) // the host's store, the host's rules + * return VaultedFailureDetail(handle, expiresAt) + * } + * + * // The host's nightly retention job deletes vault rows whose expiresAt has passed. + * // DICE is not involved in any line above. + * ``` + * + * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. + * + * @property handle The host's opaque name for the material. It identifies a row in the host's + * vault and grants no access to it. + * @property expiresAt When the host's retention job is expected to have removed the material + */ +@ApiStatus.Experimental +interface ProtectedContentRef { + + /** The host's opaque name for the material. */ + val handle: String + + /** When the host's own retention job is expected to have removed the material. */ + val expiresAt: Instant +} diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionFailureVocabularyTest.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionFailureVocabularyTest.kt new file mode 100644 index 00000000..f688d6b4 --- /dev/null +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionFailureVocabularyTest.kt @@ -0,0 +1,212 @@ +/* + * Copyright 2024-2026 Embabel Pty Ltd. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package com.embabel.dice.proposition.extraction + +import org.assertj.core.api.Assertions.assertThat +import org.junit.jupiter.api.Test +import java.io.File +import java.lang.reflect.Executable +import java.lang.reflect.Field +import java.lang.reflect.Modifier +import java.time.Instant + +/** + * The canaries: four kinds of text that must have no way into a stored failure record. + * + * Each of [LEAK_CANDIDATES] is a string, and `ExtractionFailure` has no parameter, property or + * field anywhere in its reachable shape that will take one. So every canary here is a shape + * assertion: the compiler already rejects the call, and these tests pin the shape the compiler is + * reading, so an edit that adds a text field back turns one of them red. + * + * **The lines that no longer compile.** Written out because a shape assertion is easy to read as + * abstract, and these are the concrete calls it forbids: + * + * ```kotlin + * ExtractionFailure(ExtractionFailureCode.DECODE_FAILED, ExtractionRunFixtures.SOURCE_TEXT) + * ExtractionFailure.of(ExtractionFailureCode.INTERNAL, "Extract every claim about: ") + * ExtractionFailure(ExtractionFailureCode.INTERNAL, "marguerite.okonkwo@acme-holdings.example") + * ExtractionFailure.of(ExtractionFailureCode.INTERNAL, "AKIAIOSFODNN7EXAMPLE") + * ExtractionFailure.fromThrowable(ExtractionFailureCode.DECODE_FAILED, providerException) + * ``` + * + * The first four have no string-shaped parameter to land in and the fifth has no throwable-shaped + * one, which is the whole of the closure: a caller holding something regrettable finds nowhere on + * this type to put it. + */ +class ExtractionFailureVocabularyTest { + + @Test + fun `nothing on a failure record accepts any of the canary strings`() { + val entryPoints = entryPointsOf(ExtractionFailure::class.java, ExtractionFailureMeasure::class.java) + + LEAK_CANDIDATES.forEach { (what, value) -> + val accepting = entryPoints + .filter { entry -> entry.parameterTypes.any { it.isAssignableFrom(value.javaClass) } } + .map { "${it.declaringClass.simpleName}.${it.name}" } + + assertThat(accepting).describedAs("entry points accepting %s", what).isEmpty() + } + } + + @Test + fun `nothing on a failure record accepts a throwable`() { + // A throwable is a bag of strings: the provider's message, the fragment that failed to + // parse, the stack. An API that took one and promised to read almost none of it is the + // contract that erodes on the next edit, so there is no such API. + val accepting = entryPointsOf(ExtractionFailure::class.java, ExtractionFailureMeasure::class.java) + .filter { entry -> entry.parameterTypes.any { Throwable::class.java.isAssignableFrom(it) } } + .map { "${it.declaringClass.simpleName}.${it.name}" } + + assertThat(accepting).isEmpty() + } + + @Test + fun `no type a failure record reaches has a text field`() { + // Walks the whole reachable shape, so a text field added to any type a failure holds shows + // up here even when ExtractionFailure itself stays clean. + reachableFieldTypes(ExtractionFailure::class.java).forEach { type -> + assertThat(CharSequence::class.java.isAssignableFrom(type)) + .describedAs("%s is text", type.name) + .isFalse() + } + } + + @Test + fun `every value a populated failure holds is an enum, a number, or an instant`() { + val populated = ExtractionFailure( + code = ExtractionFailureCode.SCHEMA_VIOLATION, + stage = ExtractionFailureStage.SCHEMA_CHECK, + providerStatus = 422, + measure = ExtractionFailureMeasure(ExtractionFailureQuantity.TOKEN_COUNT, 4_096), + at = Instant.parse("2026-08-31T10:15:47Z"), + invocation = ExtractionInvocationId(invocationIndex = 3, attempt = 2), + ) + + val held = reachableValues(populated) + + assertThat(held).isNotEmpty() + held.forEach { value -> + assertThat(value is Enum<*> || value is Number || value is Instant) + .describedAs("%s is a vocabulary value", value) + .isTrue() + } + } + + @Test + fun `the closed vocabulary is small enough to read`() { + // The failure record says everything it says with these. A code that keeps landing on + // UNCLASSIFIED, or a stage nobody can pick, is the signal to add a value here — and adding + // one is a reviewed change to a fixed list. + assertThat(ExtractionFailureCode.entries).hasSize(11) + assertThat(ExtractionFailureStage.entries).hasSize(10) + assertThat(ExtractionFailureQuantity.entries).hasSize(7) + + // Every quantity names its unit, so 30000 can never be read as seconds by one caller and + // millis by another. + assertThat(ExtractionFailureQuantity.entries.map { it.name }) + .allMatch { it.endsWith("_COUNT") || it.endsWith("_MILLIS") || it.endsWith("_SECONDS") } + } + + @Test + fun `the protected content reference ships as specification only`() { + // It is there so a host writing its own detail vault has a written contract to work from. + // DICE implementing one would make DICE the writer, the reader and the retention owner, + // which is the whole of what the contract hands to the host. + assertThat(ProtectedContentRef::class.java.isInterface).isTrue() + assertThat(ProtectedContentRef::class.java.declaredMethods) + .allMatch { Modifier.isAbstract(it.modifiers) } + + val mentions = compiledMainClasses() + .filter { it.name.substringBefore('.') != "ProtectedContentRef" } + .filter { String(it.readBytes(), Charsets.ISO_8859_1).contains("ProtectedContentRef") } + .map { it.name } + + // A class file mentions a type it implements, holds, or calls. KDoc pointing a reader at + // the specification compiles to nothing, so this stays empty while the docs still link it. + assertThat(mentions).isEmpty() + } + + /** Every compiled class of the DICE main source set, read as bytes. */ + private fun compiledMainClasses(): List { + val root = File(ExtractionFailure::class.java.protectionDomain.codeSource.location.toURI()) + return root.walkTopDown().filter { it.isFile && it.extension == "class" }.toList() + } + + /** Constructors and public methods: every way a caller can hand a value in. */ + private fun entryPointsOf(vararg types: Class<*>): List = + types.flatMap { type -> + val nested = type.declaredClasses.flatMap { it.declaredMethods.asList() } + (type.declaredConstructors.asList() + type.declaredMethods.asList() + nested) + .filter { Modifier.isPublic(it.modifiers) && !it.isSynthetic } + // equals takes Object, which every value is assignable to, so it says nothing here. + .filter { it.name != "equals" } + } + + /** Field types reachable from [root], recursing into DICE's own types. */ + private fun reachableFieldTypes(root: Class<*>): Set> { + val seen = mutableSetOf>() + val types = mutableSetOf>() + fun walk(type: Class<*>) { + if (!seen.add(type)) return + instanceFields(type).forEach { field -> + types += field.type + if (field.type.name.startsWith("com.embabel.")) walk(field.type) + } + } + walk(root) + return types + } + + /** Every non-null value reachable from [root], flattened, recursing into DICE's own types. */ + private fun reachableValues(root: Any): List { + val values = mutableListOf() + fun walk(value: Any) { + instanceFields(value.javaClass).forEach { field -> + field.isAccessible = true + val held = field.get(value) ?: return@forEach + if (held.javaClass.name.startsWith("com.embabel.") && held !is Enum<*>) { + walk(held) + } else { + values += held + } + } + } + walk(root) + return values + } + + private fun instanceFields(type: Class<*>): List = + type.declaredFields.filterNot { it.isSynthetic || Modifier.isStatic(it.modifiers) } + + companion object { + + /** + * The four kinds of text the reviewer named, each in the shape it really arrives in. + * + * The credential shapes are the ones a scanner recognises: an AWS access key id and an + * OpenAI-style project key. They earn a line of their own because a character-class check + * would let the AWS one through — it is all uppercase letters and digits — while a type + * with no string field lets neither through. + */ + private val LEAK_CANDIDATES: List> = listOf( + "raw source text" to ExtractionRunFixtures.SOURCE_TEXT, + "a prompt fragment" to "You are an extraction assistant. Extract every claim about: ", + "an email address" to "marguerite.okonkwo@acme-holdings.example", + "an AWS-shaped access key id" to "AKIAIOSFODNN7EXAMPLE", + "an OpenAI-shaped project key" to "sk-proj-8f2a10bd4c7e9013a5b6c8d2e4f60719", + ) + } +} diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunContractTest.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunContractTest.kt index 2719bd05..ee5bcd8c 100644 --- a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunContractTest.kt +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunContractTest.kt @@ -148,7 +148,7 @@ class ExtractionRunContractTest { assertThatIllegalArgumentException().isThrownBy { runWith( failures = (0..ExtractionRunLimits.MAX_FAILURES) - .map { ExtractionFailure(ExtractionFailureCode.INTERNAL, "failure $it", STARTED_AT) }, + .map { ExtractionFailure(code = ExtractionFailureCode.INTERNAL, at = STARTED_AT) }, ) }.withMessageContaining("failures") @@ -243,7 +243,11 @@ class ExtractionRunContractTest { assertThat( runWith( failures = listOf( - ExtractionFailure(ExtractionFailureCode.SOURCE_UNAVAILABLE, "source read failed", STARTED_AT), + ExtractionFailure( + code = ExtractionFailureCode.SOURCE_UNAVAILABLE, + stage = ExtractionFailureStage.SOURCE_READ, + at = STARTED_AT, + ), ), ).failures.single().invocation, ).isNull() @@ -269,7 +273,7 @@ class ExtractionRunContractTest { revisions += REVISION_ONE invocations += ExtractionInvocationRecord.planned(0) - failures += ExtractionFailure(ExtractionFailureCode.INTERNAL, "later", STARTED_AT) + failures += ExtractionFailure(code = ExtractionFailureCode.INTERNAL, at = STARTED_AT) assertThat(run.sourceRevisions).isEmpty() assertThat(run.invocations).isEmpty() diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunFixtures.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunFixtures.kt index 3d0757d4..4c7a70d4 100644 --- a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunFixtures.kt +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunFixtures.kt @@ -29,11 +29,11 @@ import java.time.Instant internal object ExtractionRunFixtures { /** - * The source text the sanitization tests extract from. + * The source text the privacy tests extract from. * * It carries a person's name, an organisation, an email address and a case number on purpose: - * those are the shapes that survive into a provider's exception message and then into a stored - * failure record if anything writes `e.message` into one. + * those are the shapes that survive into a provider's exception message and would reach a + * stored failure record if a failure had anywhere to put them. */ const val SOURCE_TEXT: String = "Marguerite Okonkwo confirmed the Q3 renewal for Acme Holdings on 12 March 2026; " + @@ -77,8 +77,9 @@ internal object ExtractionRunFixtures { /** * A run with every field populated, which is what the privacy assertions dump. * - * The failure on it is built the way DICE builds one — from a throwable whose message quotes - * [SOURCE_TEXT], exactly as a provider's would. + * Its failure is the one an extraction over [SOURCE_TEXT] would record: the provider threw the + * exception [providerFailureQuotingSource] builds, and what the run keeps is the code, the + * stage, the status, the size, and which attempt it belonged to. */ fun populatedRun(): ExtractionRun = ExtractionRun( contextId = CONTEXT, @@ -165,9 +166,11 @@ internal object ExtractionRunFixtures { ), ), failures = listOf( - ExtractionFailure.fromThrowable( + ExtractionFailure( code = ExtractionFailureCode.DECODE_FAILED, - throwable = providerFailureQuotingSource(), + stage = ExtractionFailureStage.RESPONSE_DECODE, + providerStatus = 502, + measure = ExtractionFailureMeasure(ExtractionFailureQuantity.CHARACTER_COUNT, 4_128), at = FINISHED_AT, invocation = ExtractionInvocationId(invocationIndex = 1, attempt = 2), ), diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunPrivacyTest.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunPrivacyTest.kt index 74ab7b18..f23180be 100644 --- a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunPrivacyTest.kt +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunPrivacyTest.kt @@ -31,10 +31,13 @@ import java.lang.reflect.Modifier * covered. `toString` gets its own, weaker check, because a run reaches logs that way. * * The honest limit of these tests: a value type can bound a string and restrict its characters, and - * it can refuse to read an exception message. It cannot tell a pseudonym from a username. So the - * assertions are about what the types enforce — shapes that cannot be stored, and the one path - * DICE itself uses being incapable of carrying source text — not about a claim that no host can - * ever put something regrettable in a token. + * a type with no string field at all can refuse every string. Neither can tell a pseudonym from a + * username. So the assertions are about what the types enforce — shapes that cannot be stored, and + * a failure record with nowhere to put source text — and they make no claim that a host cannot put + * something regrettable in a token. + * + * The failure vocabulary gets its own file, [ExtractionFailureVocabularyTest], which pins the + * closed shape itself. */ class ExtractionRunPrivacyTest { @@ -43,34 +46,44 @@ class ExtractionRunPrivacyTest { private val longDigitRun = Regex("\\d{9,}") @Test - fun `a failure built from a provider exception carries none of the source text`() { + fun `a provider exception quoting the source has no way into a failure record`() { // The leak this guards: extraction runs over known material, the provider throws, and its // message quotes the prompt — which is the source text — back at us. val thrown = runCatching { extractPropositionsFrom(SOURCE_TEXT) }.exceptionOrNull()!! assertThat(thrown.message).contains("Marguerite Okonkwo") assertThat(thrown.cause?.message).contains("AB-7741-XZ") - val failure = ExtractionFailure.fromThrowable(ExtractionFailureCode.DECODE_FAILED, thrown) - - assertThat(failure.detail) - .isEqualTo("java.lang.IllegalStateException <- java.lang.IllegalArgumentException") + // No constructor takes that exception, and none takes a string pulled out of it. + val constructorParameters = + ExtractionFailure::class.java.declaredConstructors.flatMap { it.parameterTypes.asList() } + assertThat(constructorParameters) + .doesNotContain(Throwable::class.java, String::class.java, CharSequence::class.java) + + // What the caller can record is the classification, and a dump of it holds no fragment. + val failure = ExtractionFailure( + code = ExtractionFailureCode.DECODE_FAILED, + stage = ExtractionFailureStage.RESPONSE_DECODE, + providerStatus = 502, + ) + val row = serializedRow(failure) SOURCE_TEXT_FRAGMENTS.forEach { fragment -> - assertThat(failure.detail).doesNotContain(fragment) + assertThat(row).doesNotContain(fragment) } + assertThat(row).contains("DECODE_FAILED", "RESPONSE_DECODE", "502") } @Test fun `no fragment of the source text survives into a stored run`() { val row = serializedRow(ExtractionRunFixtures.populatedRun()) - // The run's only failure was built from the exception above, so if any part of the message - // path leaked, one of these fragments would be in the row. + // The run's only failure records the provider exception above, so if any part of the + // message path leaked, one of these fragments would be in the row. SOURCE_TEXT_FRAGMENTS.forEach { fragment -> assertThat(row).doesNotContain(fragment) } assertThat(row).doesNotContain(SOURCE_TEXT) // The row really did reach the failure record, so the assertion above is not vacuous. - assertThat(row).contains("DECODE_FAILED", "java.lang.IllegalStateException") + assertThat(row).contains("DECODE_FAILED", "RESPONSE_DECODE", "CHARACTER_COUNT") } @Test @@ -90,8 +103,9 @@ class ExtractionRunPrivacyTest { assertThat(rendered).contains("runId=run-01JAV7Q2N4", "rootRunId=run-01JAV6M0K1", "status=FAILED") assertThat(rendered).contains("sourceRevisions=2", "invocations=2", "failures=1") - // Not in the summary: the tokens, the digests, the failure detail. - assertThat(rendered).doesNotContain("actor:7f19aa02", "sha256:6d1f0a2b", "IllegalStateException") + // Left out of the summary: the tokens, the digests, and everything on a failure past its + // count. + assertThat(rendered).doesNotContain("actor:7f19aa02", "sha256:6d1f0a2b", "RESPONSE_DECODE") SOURCE_TEXT_FRAGMENTS.forEach { fragment -> assertThat(rendered).doesNotContain(fragment) } } @@ -170,38 +184,25 @@ class ExtractionRunPrivacyTest { } @Test - fun `a caller-written failure detail is flattened and clipped, and DICE says it cannot vouch for it`() { - val wordy = "chunk 3 of 12\n exceeded the budget\t" + "x".repeat(1_000) - - val failure = ExtractionFailure.of(ExtractionFailureCode.SCHEMA_VIOLATION, wordy) - - assertThat(failure.detail).hasSize(ExtractionRunLimits.MAX_FAILURE_DETAIL_LENGTH) - assertThat(failure.detail).startsWith("chunk 3 of 12 exceeded the budget x") - assertThat(failure.detail).doesNotContain("\n") - - // The constructor is stricter than the factory: it rejects rather than clipping, so a - // failure record can never be constructed over the bound by accident. - assertThatIllegalArgumentException().isThrownBy { - ExtractionFailure(ExtractionFailureCode.SCHEMA_VIOLATION, "y".repeat(513)) - } - assertThatIllegalArgumentException().isThrownBy { - ExtractionFailure(ExtractionFailureCode.SCHEMA_VIOLATION, "two\nlines") - }.withMessageContaining("single line") - } - - @Test - fun `a cause chain is bounded and a self-referencing cause terminates`() { - val deep = (1..12).fold(RuntimeException("root") as Throwable) { cause, _ -> - IllegalStateException("wrapper", cause) - } - val failure = ExtractionFailure.fromThrowable(ExtractionFailureCode.INTERNAL, deep) - - assertThat(failure.detail.split(" <- ")).hasSize(ExtractionFailure.MAX_CAUSE_CHAIN) + fun `the numbers on a failure are checked, and a rejection quotes only the number`() { + val failure = ExtractionFailure( + code = ExtractionFailureCode.RATE_LIMITED, + providerStatus = 429, + measure = ExtractionFailureMeasure(ExtractionFailureQuantity.RETRY_AFTER_SECONDS, 30), + ) - val selfCausing = SelfCausingException() - assertThat( - ExtractionFailure.fromThrowable(ExtractionFailureCode.INTERNAL, selfCausing).detail, - ).isEqualTo(SelfCausingException::class.java.name) + assertThat(failure.providerStatus).isEqualTo(429) + assertThat(failure.measure?.value).isEqualTo(30) + + assertThatIllegalArgumentException() + .isThrownBy { ExtractionFailure(ExtractionFailureCode.RATE_LIMITED, providerStatus = 99) } + .withMessageContaining("HTTP status") + assertThatIllegalArgumentException() + .isThrownBy { ExtractionFailure(ExtractionFailureCode.RATE_LIMITED, providerStatus = 600) } + .withMessageContaining("HTTP status") + assertThatIllegalArgumentException() + .isThrownBy { ExtractionFailureMeasure(ExtractionFailureQuantity.TOKEN_COUNT, -1) } + .withMessageContaining("TOKEN_COUNT") } @Test @@ -234,10 +235,6 @@ class ExtractionRunPrivacyTest { IllegalArgumentException("unexpected token near '$sourceText'"), ) - private class SelfCausingException : RuntimeException("self") { - override val cause: Throwable get() = this - } - /** * Renders every field of a value, recursively, the way a row writer would see it. * diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunValueTypesTest.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunValueTypesTest.kt index 8b9ee367..6bec56ff 100644 --- a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunValueTypesTest.kt +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunValueTypesTest.kt @@ -169,17 +169,26 @@ class ExtractionRunValueTypesTest { val bare = ExtractionFailure(ExtractionFailureCode.MODEL_TIMEOUT) assertThat(bare.code).isEqualTo(ExtractionFailureCode.MODEL_TIMEOUT) - assertThat(bare.detail).isEmpty() + assertThat(bare.stage).isNull() + assertThat(bare.providerStatus).isNull() + assertThat(bare.measure).isNull() assertThat(bare.invocation).isNull() + // "the provider's quota ran out on the second try at call 2" said in the vocabulary. val tied = ExtractionFailure.of( code = ExtractionFailureCode.RATE_LIMITED, - detail = "provider quota exhausted", + stage = ExtractionFailureStage.MODEL_CALL, + providerStatus = 429, + measure = ExtractionFailureMeasure(ExtractionFailureQuantity.RETRY_AFTER_SECONDS, 60), at = ExtractionRunFixtures.FINISHED_AT, invocation = ExtractionInvocationId(2, 3), ) assertThat(tied.invocation).isEqualTo(ExtractionInvocationId(2, 3)) - assertThat(tied.detail).isEqualTo("provider quota exhausted") + assertThat(tied.stage).isEqualTo(ExtractionFailureStage.MODEL_CALL) + assertThat(tied.providerStatus).isEqualTo(429) + assertThat(tied.measure) + .isEqualTo(ExtractionFailureMeasure.of(ExtractionFailureQuantity.RETRY_AFTER_SECONDS, 60)) + assertThat(tied.measure.toString()).isEqualTo("RETRY_AFTER_SECONDS=60") } @Test @@ -204,6 +213,10 @@ class ExtractionRunValueTypesTest { ExtractionProviderResponseFacts::class.java, ExtractionFailure::class.java, ExtractionFailureCode::class.java, + ExtractionFailureStage::class.java, + ExtractionFailureQuantity::class.java, + ExtractionFailureMeasure::class.java, + ProtectedContentRef::class.java, ExtractionOpaqueRef::class.java, ExtractionActorRef::class.java, ExtractionRequestRef::class.java, diff --git a/docs/design/extraction-runs.md b/docs/design/extraction-runs.md index c2183654..240f4de4 100644 --- a/docs/design/extraction-runs.md +++ b/docs/design/extraction-runs.md @@ -26,7 +26,8 @@ flowchart TD RUN --> EXP["ExtractionExperimentRef
ExtractionCohortRef"] RUN --> ST["ExtractionRunStatus
ExtractionReplayFidelity
ExtractionRunCounts"] RUN --> INV["List<ExtractionInvocationRecord>
what was observed"] - RUN --> FAIL["List<ExtractionFailure>
bounded, sanitized"] + RUN --> FAIL["List<ExtractionFailure>
bounded, closed vocabulary"] + FAIL --> FV["ExtractionFailureCode
ExtractionFailureStage
ExtractionFailureMeasure"] INV --> IID["ExtractionInvocationId
plan ordinal + attempt"] INV --> USE["ExtractionModelUsage"] INV --> PRF["ExtractionProviderResponseFacts"] @@ -173,29 +174,76 @@ characters, so a reference does not spread through logs in full. And a validatio field and the length and never quotes the value — an `IllegalArgumentException` propagates into logs, and the value that failed validation is exactly the one nobody vouched for. -### Failures +### Failures speak a closed vocabulary -A failure record is a classified code plus a short single-line detail, and the run holds at most 64 -of them. +A failure record is a classified code, an optional stage, an optional provider status, an optional +number with its unit, a timestamp, and the attempt it belongs to. The run holds at most 64 of them. +There is no text field anywhere in that list. Failure records are where source text leaks. A provider quotes the prompt back in its exception message; a decode error carries the fragment it choked on. Both land in a stored run header the -moment someone writes `e.message` into one. So `ExtractionFailure.fromThrowable` — the path DICE -itself uses — never reads `Throwable.message`. It records the exception class names down the cause -chain, bounded to five links and cycle-safe, and nothing else. That detail cannot contain source -text because it never touched any. - -`ExtractionFailure.of` exists for the case where a caller genuinely knows something useful -("chunk 3 of 12 exceeded the token budget"). DICE cannot check what a caller puts there; it bounds -it and flattens it to a single line so a pasted stack trace does not fit. - -The tests are built to match what is actually enforced. A fixture with known source text -(a person, an organisation, an email address, a case number) is fed through a provider-shaped -exception that quotes it; the resulting failure and a full field-by-field dump of the populated run -are asserted to contain none of its fragments, no address shape, no link shape, and no long digit -run. The dump is reflective rather than `toString`, so a field the summary omits is still covered — -and `run.toString()` gets its own check that it shows identity, state and sizes and none of the -tokens, digests or details. +moment someone writes `e.message` into a text field. A #95 review comment found that the earlier +shape — a code plus a bounded, whitespace-flattened `detail` string — closed nothing: truncating a +prompt still stores a prompt, and a credential, an email address or a paragraph of protected text +all fit in 512 characters. The vocabulary closes it at construction. `ExtractionFailure` has no +`String` parameter, property or field, and no factory that takes a `Throwable`, so free text has no +route into durable storage through this type. + +What the vocabulary carries: + +| Field | Type | Answers | +| --- | --- | --- | +| `code` | `ExtractionFailureCode` (11 values) | what went wrong | +| `stage` | `ExtractionFailureStage` (10 values) | where in the run's work | +| `providerStatus` | `Int?`, 100..599 | what the provider returned | +| `measure` | `ExtractionFailureMeasure` | one "how much", with its unit | +| `invocation` | `ExtractionInvocationId?` | which call and which attempt | + +`ExtractionFailureMeasure` pairs a number with an `ExtractionFailureQuantity` naming both the thing +and its unit — `TOKEN_COUNT`, `ELAPSED_MILLIS`, `RETRY_AFTER_SECONDS`. A bare number on a failure +record is a unit-mismatch bug waiting to happen, and pairing them in a type means 4096 can never be +recorded without saying it is tokens. One measure per record: a failure answers one "how much". + +"Chunk 3 of 12 exceeded the token budget" survives the change, said in the vocabulary — the chunk +is `invocation.invocationIndex`, which the call plan already allocated, and the budget is a +`TOKEN_COUNT` measure. What is lost is the sentence, which is the part nobody could vouch for. + +A host that wants the exception message, the response body, or the fragment that failed to parse +keeps that material itself. `ProtectedContentRef` is the written contract for the reference such a +host passes around; see below. + +The tests match what is enforced. `ExtractionFailureVocabularyTest` takes four kinds of text a +reviewer named — raw source text, a prompt fragment, an email address, and two credential shapes a +scanner recognises — and asserts no constructor, factory or method on the type will take any of +them, that no type a failure reaches has a text field, and that every value a fully populated +failure holds is an enum, a number or an instant. The privacy suite still feeds a fixture with +known source text (a person, an organisation, an email address, a case number) through a +provider-shaped exception that quotes it, and asserts a full field-by-field dump of the populated +run contains none of its fragments, no address shape, no link shape, and no long digit run. The +dump is reflective, so a field the summary omits is still covered — and +`run.toString()` gets its own check that it shows identity, state and sizes and none of the tokens, +digests or failure fields. + +### The protected reference is a specification + +`ProtectedContentRef` is an interface with two members — an opaque `handle` and an `expiresAt` — +and no implementation anywhere in DICE. It is the written contract for a host that keeps detailed +failure material of its own. + +The host owns all three jobs. The **writer** is host code, because DICE never sees the material. +The **reader** is host code, because resolving a handle is a host operation under the host's access +rules; the handle names a row in the host's vault and grants no access to it, so a signed URL, a +bearer token or a decryption key breaks the contract. **Retention** is the host's, and `expiresAt` +is where the host writes it down — nothing in DICE sweeps, deletes, or checks it, and an erasure +request reaches the material through the host's vault. + +A type of this name shipped in an earlier #98 draft as a stored value and was deleted during +review, because nothing attached it to a run and DICE had no writer, no reader and no retention +behaviour behind it. It returns as specification only, which is what it always was. The KDoc +carries a worked example: a host minting a handle, writing an exception message into its own vault +under it for ninety days, and running its own nightly retention job. A test asserts the interface +is abstract and that no compiled DICE class mentions the type, so "zero implementations, zero +production references" is a checked property. ## Replay is approximate, and named that way @@ -255,14 +303,14 @@ different id, and a store would then key rows on a value the caller never minted | Constant | Value | Applies to | | --- | --- | --- | | `MAX_IDENTIFIER_LENGTH` | 256 | opaque tokens, fingerprints, model and role names, service names, provider response ids, runtime identifiers | -| `MAX_FAILURE_DETAIL_LENGTH` | 512 | the one free-text field | +| `MIN_PROVIDER_STATUS` / `MAX_PROVIDER_STATUS` | 100 / 599 | the status a failure records | | `MAX_SOURCE_REVISIONS` | 256 | source revisions per run | | `MAX_INVOCATIONS` | 1024 | invocation records per run, across every call and attempt | | `MAX_FAILURES` | 64 | failure records per run | -The failure detail is the one exception to rejection, and only on the way in: the factories shorten -it before construction, because keeping a clipped failure record beats losing the failure. The -constructor still rejects a longer one. +The rule has no exception now that the model has no free-text field for one to apply to. The +failure detail used to be one, clipped by its factories on the way in; the closed vocabulary +removed the field and the exception with it. Lengths count UTF-16 chars, so a 256-char identifier can be around 1 KB of UTF-8. The bound exists to keep a run header finite. From 1556754cdb70c670dbeae45a0cfa5b5012f69dca Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Wed, 2 Sep 2026 09:56:17 -0400 Subject: [PATCH 4/7] State the metamodel join in the run design doc The decision lived in envelope KDoc only. The design doc now says what the versioning doc says in the same words: the run record carries the declared schema's content hash, the coordinator resolves it from the host's DeclaredSchemaSource, and per-proposition attribution answers through run lineage. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com> --- docs/design/extraction-runs.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/design/extraction-runs.md b/docs/design/extraction-runs.md index 240f4de4..82b19e94 100644 --- a/docs/design/extraction-runs.md +++ b/docs/design/extraction-runs.md @@ -39,6 +39,24 @@ flowchart TD assumes it is unique across tenants, so the `ContextId` travels with it. Two tenants that both mint `run-1` have two runs. +## How a run joins to the metamodel + +One field carries the join. `ExtractionRunFingerprints.metamodelFingerprint` holds the declared +schema's content hash — the `contentHash` a `MetamodelVersion` fingerprints itself with, which +`docs/design/metamodel-versioning.md` on the metamodel stack describes. DICE compares it and stores +it and reads nothing out of it, the same as the other two fingerprints. + +The extraction coordinator writes it, and that coordinator is a later slice. It resolves the hash +from the host's `DeclaredSchemaSource` and stamps it once per run. Nothing in this slice produces +one, so a run built today carries whatever its caller passed. + +The fingerprint says what the whole run ran under. A proposition's schema attribution is answered +through the run that produced it (`PRODUCED_BY_RUN`, the proposition-to-run relation a later slice +adds); the run record carries the declared schema's content hash, resolved by the extraction +coordinator from the host's `DeclaredSchemaSource`. That is why no per-proposition schema stamp +exists. A per-proposition denormalized copy is a coordinator concern for a slice whose reads demand +it. + ## Requested and observed are different types The single most useful thing a run can tell an incident is which of these two it is looking at: From b56b3fcb83c3ad6641aad199135b62906207f130 Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Wed, 9 Sep 2026 17:05:09 -0400 Subject: [PATCH 5/7] Speak LlmHyperparameters from the requested model config ExtractionRequestedModelConfig already carried the same six hyperparameters the framework's LlmOptions does, under the same names. It now implements LlmHyperparameters directly, so a caller holding the record already holds something that reads as the interface, not a lookalike copy. from(options, modelRole, thinkingFingerprint, selectionFingerprint) builds a config straight off an LlmOptions a host handed the model, taking modelRole from options.role by default. No field is renamed and the constructor descriptor is unchanged; the six fields just gain override. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com> --- CHANGELOG.md | 8 +++ .../ExtractionRequestedModelConfig.kt | 51 ++++++++++++++++--- .../extraction/ExtractionRunValueTypesTest.kt | 37 ++++++++++++++ docs/design/extraction-runs.md | 3 ++ 4 files changed, 92 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 93894060..9ee38b24 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1329,3 +1329,11 @@ and the consumer PRs that deliver it). asserted by a test that reads the class files because the annotation has class retention, and the shapes may still move while the remaining #67 slices land. The `@RequiresOptIn` question #66 raised is unchanged and still open. + +- `ExtractionRequestedModelConfig` now implements the framework's `LlmHyperparameters` (PR #95 + review), so the six hyperparameters it already carried read back as the interface a host that + built an `LlmOptions` already knows, not as a lookalike copy. `from(options, modelRole, + thinkingFingerprint, selectionFingerprint)` builds one straight off an `LlmOptions`, taking + `modelRole` from `options.role` by default. **Compatibility: additive.** The data class's + constructor descriptor is unchanged; the six fields keep their names and types and just gain + `override`, and the new supertype adds no field. diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRequestedModelConfig.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRequestedModelConfig.kt index 2a00767a..eeb8880f 100644 --- a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRequestedModelConfig.kt +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionRequestedModelConfig.kt @@ -15,6 +15,8 @@ */ package com.embabel.dice.proposition.extraction +import com.embabel.common.ai.model.LlmHyperparameters +import com.embabel.common.ai.model.LlmOptions import org.jetbrains.annotations.ApiStatus import java.time.Duration @@ -39,6 +41,11 @@ import java.time.Duration * no upper bound here because services differ on whether it stops at 1 or 2; the penalties are * only required to be real numbers for the same reason. * + * The six hyperparameters are the framework's own [LlmHyperparameters], not a lookalike copy, so a + * caller holding this record already holds something that reads as one. [from] builds one straight + * from an [LlmOptions] a host handed the model, and the record hands the same six values back + * through that interface. + * * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. * * @property modelRole The host's name for the job this model was doing, such as `extraction` @@ -57,16 +64,16 @@ import java.time.Duration data class ExtractionRequestedModelConfig @JvmOverloads constructor( val modelRole: String? = null, val requestedModel: String? = null, - val temperature: Double? = null, - val topP: Double? = null, - val topK: Int? = null, - val maxTokens: Int? = null, - val presencePenalty: Double? = null, - val frequencyPenalty: Double? = null, + override val temperature: Double? = null, + override val topP: Double? = null, + override val topK: Int? = null, + override val maxTokens: Int? = null, + override val presencePenalty: Double? = null, + override val frequencyPenalty: Double? = null, val thinkingFingerprint: String? = null, val selectionFingerprint: String? = null, val timeout: Duration? = null, -) { +) : LlmHyperparameters { init { requireBoundedIdentifier(modelRole, "modelRole") @@ -114,5 +121,35 @@ data class ExtractionRequestedModelConfig @JvmOverloads constructor( selectionFingerprint = selectionFingerprint, timeout = timeout, ) + + /** + * The record of what a host asked for, read straight off the [LlmOptions] it handed the + * model. + * + * The six hyperparameters and the model and timeout come from [options] as they stand. + * [modelRole] defaults to [LlmOptions.role] because that is the closest thing the options + * carry to a job name, but a host is free to pass its own. The two fingerprints are not on + * [LlmOptions] at all, so a caller that has them from elsewhere passes them in here. + */ + @JvmStatic + @JvmOverloads + fun from( + options: LlmOptions, + modelRole: String? = options.role, + thinkingFingerprint: String? = null, + selectionFingerprint: String? = null, + ): ExtractionRequestedModelConfig = ExtractionRequestedModelConfig( + modelRole = modelRole, + requestedModel = options.model, + temperature = options.temperature, + topP = options.topP, + topK = options.topK, + maxTokens = options.maxTokens, + presencePenalty = options.presencePenalty, + frequencyPenalty = options.frequencyPenalty, + thinkingFingerprint = thinkingFingerprint, + selectionFingerprint = selectionFingerprint, + timeout = options.timeout, + ) } } diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunValueTypesTest.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunValueTypesTest.kt index 6bec56ff..e6352697 100644 --- a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunValueTypesTest.kt +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunValueTypesTest.kt @@ -15,6 +15,8 @@ */ package com.embabel.dice.proposition.extraction +import com.embabel.common.ai.model.LlmHyperparameters +import com.embabel.common.ai.model.LlmOptions import org.assertj.core.api.Assertions.assertThat import org.assertj.core.api.Assertions.assertThatIllegalArgumentException import org.junit.jupiter.api.Test @@ -72,6 +74,41 @@ class ExtractionRunValueTypesTest { } } + @Test + fun `the requested model config is the framework's hyperparameters`() { + val options = LlmOptions( + model = "model-large", + role = "extraction", + temperature = 0.4, + frequencyPenalty = 0.1, + maxTokens = 2048, + presencePenalty = 0.2, + topK = 40, + topP = 0.9, + timeout = Duration.ofSeconds(30), + ) + + val config = ExtractionRequestedModelConfig.from(options) + + assertThat(config.modelRole).isEqualTo("extraction") + assertThat(config.requestedModel).isEqualTo("model-large") + assertThat(config.temperature).isEqualTo(0.4) + assertThat(config.frequencyPenalty).isEqualTo(0.1) + assertThat(config.maxTokens).isEqualTo(2048) + assertThat(config.presencePenalty).isEqualTo(0.2) + assertThat(config.topK).isEqualTo(40) + assertThat(config.topP).isEqualTo(0.9) + assertThat(config.timeout).isEqualTo(Duration.ofSeconds(30)) + + val asHyperparameters: LlmHyperparameters = config + assertThat(asHyperparameters.temperature).isEqualTo(options.temperature) + assertThat(asHyperparameters.frequencyPenalty).isEqualTo(options.frequencyPenalty) + assertThat(asHyperparameters.maxTokens).isEqualTo(options.maxTokens) + assertThat(asHyperparameters.presencePenalty).isEqualTo(options.presencePenalty) + assertThat(asHyperparameters.topK).isEqualTo(options.topK) + assertThat(asHyperparameters.topP).isEqualTo(options.topP) + } + @Test fun `temperature has no upper bound because providers disagree on one`() { // Some services stop at 1 and some at 2. Rejecting a legitimate 2.0 would be DICE deciding diff --git a/docs/design/extraction-runs.md b/docs/design/extraction-runs.md index 82b19e94..e281f8fa 100644 --- a/docs/design/extraction-runs.md +++ b/docs/design/extraction-runs.md @@ -88,6 +88,9 @@ credentials, system prompts and whole SDK request bodies get persisted by accide provider-specific knob a host cares about is folded into one of the fingerprints — an opaque digest DICE compares and never reads. +The six hyperparameters implement the framework's own `LlmHyperparameters`, and `from(LlmOptions)` +builds the record straight off the options a host handed the model. + Ranges are checked where every provider agrees and left open where they do not. Temperature has no upper bound because services differ on whether it stops at 1 or 2, and the penalties are only required to be real numbers for the same reason. Rejecting a legitimate `2.0` would be DICE From 65d44c43f53d5144e9160c1878af71d17e5bd621 Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Wed, 9 Sep 2026 17:06:26 -0400 Subject: [PATCH 6/7] Build a usage record from the framework's Usage ExtractionModelUsage gains a companion factory, from(usage), that reads a call's counts straight off the framework's Usage: promptTokens becomes inputTokens, completionTokens becomes outputTokens, and totalTokens carries across unchanged. cachedInputTokens and reasoningTokens stay null, because Usage does not report either one. The record stays its own type. Usage is a final class carrying a native SDK usage object this record deliberately never stores, so extending it is not an option and converting from it is. Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com> --- CHANGELOG.md | 7 ++++++ .../extraction/ExtractionInvocationRecord.kt | 20 +++++++++++++++++ .../extraction/ExtractionRunValueTypesTest.kt | 22 +++++++++++++++++++ docs/design/extraction-runs.md | 4 ++++ 4 files changed, 53 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9ee38b24..9bf13348 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1337,3 +1337,10 @@ and the consumer PRs that deliver it). `modelRole` from `options.role` by default. **Compatibility: additive.** The data class's constructor descriptor is unchanged; the six fields keep their names and types and just gain `override`, and the new supertype adds no field. + +- `ExtractionModelUsage` gains a companion factory, `from(usage: com.embabel.agent.core.Usage)` + (PR #95 review), mapping `promptTokens` to `inputTokens`, `completionTokens` to `outputTokens` + and `totalTokens` straight across, and leaving `cachedInputTokens` and `reasoningTokens` null + because core `Usage` does not report them. The record stays its own type: `Usage` is final and + carries the native SDK object this record deliberately does not store. **Compatibility: + additive.** One new factory method, nothing existing changes. diff --git a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionInvocationRecord.kt b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionInvocationRecord.kt index d4e03106..979f5c07 100644 --- a/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionInvocationRecord.kt +++ b/dice/src/main/kotlin/com/embabel/dice/proposition/extraction/ExtractionInvocationRecord.kt @@ -15,6 +15,7 @@ */ package com.embabel.dice.proposition.extraction +import com.embabel.agent.core.Usage import org.jetbrains.annotations.ApiStatus import java.time.Instant @@ -95,6 +96,11 @@ enum class ExtractionInvocationOutcome { * Native usage objects are not stored. These are the portable counts, pulled out of whatever * shape the SDK returned. * + * This is not the framework's own [Usage], on purpose: [Usage] is a final class that cannot be + * extended, it carries a native SDK usage object this record never stores, and it has no field for + * cached or reasoning tokens. [from] converts the three counts [Usage] does carry and leaves the + * other two null. + * * EXPERIMENTAL. The shape may still change while extraction runs (DICE #67) land. * * @property inputTokens Tokens the provider counted on the way in @@ -138,6 +144,20 @@ data class ExtractionModelUsage @JvmOverloads constructor( cachedInputTokens = cachedInputTokens, reasoningTokens = reasoningTokens, ) + + /** + * The counts a call actually consumed, read off the framework's own [Usage]. + * + * [Usage.promptTokens] becomes [inputTokens], [Usage.completionTokens] becomes + * [outputTokens], and [Usage.totalTokens] carries straight across. [cachedInputTokens] and + * [reasoningTokens] stay null, because core [Usage] does not report either one. + */ + @JvmStatic + fun from(usage: Usage): ExtractionModelUsage = ExtractionModelUsage( + inputTokens = usage.promptTokens, + outputTokens = usage.completionTokens, + totalTokens = usage.totalTokens, + ) } } diff --git a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunValueTypesTest.kt b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunValueTypesTest.kt index e6352697..5ee7c838 100644 --- a/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunValueTypesTest.kt +++ b/dice/src/test/kotlin/com/embabel/dice/proposition/extraction/ExtractionRunValueTypesTest.kt @@ -15,6 +15,7 @@ */ package com.embabel.dice.proposition.extraction +import com.embabel.agent.core.Usage import com.embabel.common.ai.model.LlmHyperparameters import com.embabel.common.ai.model.LlmOptions import org.assertj.core.api.Assertions.assertThat @@ -133,6 +134,27 @@ class ExtractionRunValueTypesTest { ).forEach { construct -> assertThatIllegalArgumentException().isThrownBy { construct() } } } + @Test + fun `a usage record is built from the framework's Usage`() { + val set = Usage(promptTokens = 100, completionTokens = 20, nativeUsage = null) + val usage = ExtractionModelUsage.from(set) + + assertThat(usage.inputTokens).isEqualTo(100) + assertThat(usage.outputTokens).isEqualTo(20) + assertThat(usage.totalTokens).isEqualTo(120) + assertThat(usage.cachedInputTokens).isNull() + assertThat(usage.reasoningTokens).isNull() + + val nullCounts = Usage(promptTokens = null, completionTokens = null, nativeUsage = null) + val nullUsage = ExtractionModelUsage.from(nullCounts) + + assertThat(nullUsage.inputTokens).isNull() + assertThat(nullUsage.outputTokens).isNull() + assertThat(nullUsage.totalTokens).isNull() + assertThat(nullUsage.cachedInputTokens).isNull() + assertThat(nullUsage.reasoningTokens).isNull() + } + @Test fun `provider response facts stay absent rather than being filled in from the request`() { val absent = ExtractionProviderResponseFacts() diff --git a/docs/design/extraction-runs.md b/docs/design/extraction-runs.md index e281f8fa..322d8c2f 100644 --- a/docs/design/extraction-runs.md +++ b/docs/design/extraction-runs.md @@ -96,6 +96,10 @@ upper bound because services differ on whether it stops at 1 or 2, and the penal required to be real numbers for the same reason. Rejecting a legitimate `2.0` would be DICE deciding for a provider it never talks to. +`ExtractionModelUsage.from(usage)` converts from the framework's own `Usage`, mapping its three +counts across and leaving `cachedInputTokens` and `reasoningTokens` null, because `Usage` does not +carry them and the record stays its own type, not a `Usage` subclass. + ## Invocation identity comes from the plan, never from completion A run makes zero, one, or many model calls — chunking splits work, retries repeat it. Every From 687e61e0516287cd8863d7dae330ad178846a251 Mon Sep 17 00:00:00 2001 From: James Dunnam <7660553+jimador@users.noreply.github.com> Date: Wed, 9 Sep 2026 17:07:34 -0400 Subject: [PATCH 7/7] Say how a run relates to a host episode audit Signed-off-by: James Dunnam <7660553+jimador@users.noreply.github.com> --- docs/design/extraction-runs.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/design/extraction-runs.md b/docs/design/extraction-runs.md index 322d8c2f..f2defb3b 100644 --- a/docs/design/extraction-runs.md +++ b/docs/design/extraction-runs.md @@ -6,6 +6,12 @@ model for, what the provider actually reported back, how far it got, and what we holds none of the material. No prompts, no source text, no responses, no user or session objects, no provider SDK payloads, no extension maps. +A run is the producer-side record of an extraction. A host that keeps its own audit of what +happened in a session, an episode audit say, is looking at the same event from the consumer side: +what ran, with what, and what came out. DICE does not model the host's episode; it writes the run +it can know about and leaves `profile` and the run lineage as the join points a host uses to fold +runs into its own audit. + This note covers DICE #67's value model — the types in `com.embabel.dice.proposition.extraction` that later slices store, key, and expose. The lifecycle state machine, the store contract, the Drivine implementation, the proposition-to-run relation and the wiring are separate slices; where