Fit optional native Score cuts and Choice weights on calibration

README.md

@@ -59,7 +59,7 @@ budget-bounded, frozen artifact** that an application can run without DSPy.
 | Concern | DSPy 3.4 (experimental, per its docs and PRs) | Typewright |
 | --- | --- | --- |
 | Use Jev in a Python program | Yes: native client, async, and generative LMs mixed in one program. | Not its job. Jev only (mock and TypeSafe backends), no async. |
-| Tune decision knobs | `ReAnchor` fits Noul thresholds, Score cuts, and Choice weights. | Fits Noul thresholds and review gates. Score cuts and Choice weights are planned ([#92](https://github.com/Jev-Engineering/TypeWright/issues/92)). |
+| Tune decision knobs | `ReAnchor` fits Noul thresholds, Score cuts, and Choice weights. | Fits Noul thresholds and review gates; flat `compile --fit-decision-knobs` opts in to native Score cuts and Choice selection weights from cached calibration predictions. |
 | Search wording and structure | GEPA over instructions. Program decomposition (via generated code) is in an open upstream PR. | GEPA plus bounded structural proposals as validated JSON. No generated code. |
 | Held-out discipline | You supply the data; `ReAnchor` uses folds. | Four disjoint splits with leakage checks; test only after freeze. |
 | Review and abstention | None documented. | Fitted review gates with a minimum accepted count and an error constraint; `review_required` on every output. |
@@ -365,15 +365,17 @@ comparisons, paired cluster inference, and replayable evidence.
 **Implemented:** typed declarations and JSON artifacts; a native TypeSafe SDK
 adapter; DSPy synthesis and mutation; a standalone GEPA adapter; bounded
 add/drop/revise and numeric-decomposition proposals; validation-based selection;
-separate empirical policy fitting; projected inputs; request budgets; an optional
+separate empirical policy fitting, including optional native Score cuts and Choice
+selection weights from cached calibration predictions with flat
+`compile --fit-decision-knobs` (off by default); projected inputs; request budgets; an optional
 sensitive-output cache; reports; schema export; hardening proposals; a no-key
 fixture; the hierarchy v1 source and frozen-graph format with strict evidence
 replay; and a preregistered study runner.
 
 **Not claimed:** measured hierarchy accuracy gains; a completed independent live
 study; production certification; general multi-stage agent orchestration;
 arbitrary code generation; MIPRO integration; learned posterior calibration; a
-supervised weight fitter; automatic release approval; a TypeScript runtime;
+supervised composition-weight fitter; automatic release approval; a TypeScript runtime;
 pretrained weights; or a hosted service.
 
 The artifact stays JSON all the way down: no `eval`, generated Python, dynamic

docs/ARCHITECTURE.md

@@ -179,6 +179,36 @@ accepted count and an empirical error constraint. If no gate qualifies, it sets
 `force_review=true`. A separate probability recalibrator such as isotonic
 regression or temperature scaling is **not implemented**.
 
+Flat `compile --fit-decision-knobs` (Python `CompileOptions(fit_decision_knobs=True)`)
+opts in to optional `native-policy/v1` extensions that fit strictly ordered `score_cuts` (K-1
+cuts in 0..K-1) or positive `choice_weights` for exactly the fixed labels. Absent
+extensions serialize exactly as older policies, preserving existing artifact
+checksums and behavior. Score cuts map the continuous native expectation (or
+rescaled composite value) to an ordinal level; Choice multipliers change only
+selection, with native ties preserved. Raw distributions, P(true), and vendor
+confidence remain separate and unchanged. Direct fitted gates use the native
+probability of the selected level or weighted-selected label, and name that
+selection provenance. Composite Score retains its explicitly heuristic minimum
+component gate; no joint probability is created.
+
+Without that opt-in, decision knobs remain unchanged and report `disabled`;
+existing Noul thresholds and review-gate fitting retain their behavior. Hierarchy
+compilation keeps its existing routing and final review gates, and rejects this
+flat-only flag.
+
+The native fitter makes no model calls and imports no DSPy. Only cached calibration
+answers and labels enter bounded two-pass coordinate search. Five grouped folds
+are constructed deterministically from IDs/groups, independent of outcomes. Each
+held-out fold needs at least `max(2, min_samples)` observations. The objective is
+Choice misclassification or normalized Score MAE. No held-out fold may worsen and
+the aggregate loss must strictly improve; ties, sparse folds, or unsuccessful
+full-calibration refits report `decision_fit.status="unchanged"` and preserve the
+original selection knobs. Review-gate fitting remains a separate operational
+step and may still set `force_review`. Reports include fold losses, train/held-out
+counts, chosen policies and synthetic provenance. These checks are conservative
+heuristics, not an independent evaluation or a claim of gains. Validation selects
+prompts; it and test data never enter fitting. Freeze still precedes test.
+
 The reported Wilson interval is descriptive; it is not a corrected guarantee
 under adaptive threshold selection or distribution shift. Per-output review flags
 are not a joint guarantee for the whole decision vector. Production needs its own

docs/ROADMAP.md

@@ -20,9 +20,11 @@ rather than capabilities claimed by the current release.
 3. **Richer policy and calibration.** Add task-level utility matrices, statistically
    justified selective-risk control, optional probability recalibration, and
    supervised composition-weight fitting with carefully isolated data. Fitting
-   Score cuts and Choice weights on the calibration split, where today only Noul
-   thresholds are fitted, is tracked in
-   [#92](https://github.com/Jev-Engineering/TypeWright/issues/92).
+   Score cuts and Choice selection weights is now implemented natively from cached
+   calibration predictions in flat programs, explicitly enabled with
+   `compile --fit-decision-knobs`, with conservative grouped five-fold checks and an
+   explicit unchanged result (#92; owner approved native fitting for #95).
+   This is not posterior calibration or a measured gain.
 4. **Scale and operations.** Async batched evaluation, explicitly authorized retry
    accounting, drift monitoring, signed artifact attestations, and a separately
    reviewed TypeScript runtime. Resumable non-pickle records already exist for

schemas/hierarchy-artifact.schema.json

@@ -466,6 +466,51 @@
           "default": false,
           "title": "Force Review",
           "type": "boolean"
+        },
+        "fitting_version": {
+          "anyOf": [
+            {
+              "const": "native-policy/v1",
+              "type": "string"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "title": "Fitting Version"
+        },
+        "score_cuts": {
+          "anyOf": [
+            {
+              "items": {
+                "minimum": 0,
+                "type": "number"
+              },
+              "type": "array"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "title": "Score Cuts"
+        },
+        "choice_weights": {
+          "anyOf": [
+            {
+              "additionalProperties": {
+                "exclusiveMinimum": 0,
+                "type": "number"
+              },
+              "type": "object"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "title": "Choice Weights"
         }
       },
       "title": "Policy",

schemas/hierarchy-source.schema.json

@@ -390,6 +390,51 @@
           "default": false,
           "title": "Force Review",
           "type": "boolean"
+        },
+        "fitting_version": {
+          "anyOf": [
+            {
+              "const": "native-policy/v1",
+              "type": "string"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "title": "Fitting Version"
+        },
+        "score_cuts": {
+          "anyOf": [
+            {
+              "items": {
+                "minimum": 0,
+                "type": "number"
+              },
+              "type": "array"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "title": "Score Cuts"
+        },
+        "choice_weights": {
+          "anyOf": [
+            {
+              "additionalProperties": {
+                "exclusiveMinimum": 0,
+                "type": "number"
+              },
+              "type": "object"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "title": "Choice Weights"
         }
       },
       "title": "Policy",

schemas/program.schema.json

@@ -154,6 +154,51 @@
           "default": false,
           "title": "Force Review",
           "type": "boolean"
+        },
+        "fitting_version": {
+          "anyOf": [
+            {
+              "const": "native-policy/v1",
+              "type": "string"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "title": "Fitting Version"
+        },
+        "score_cuts": {
+          "anyOf": [
+            {
+              "items": {
+                "minimum": 0,
+                "type": "number"
+              },
+              "type": "array"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "title": "Score Cuts"
+        },
+        "choice_weights": {
+          "anyOf": [
+            {
+              "additionalProperties": {
+                "exclusiveMinimum": 0,
+                "type": "number"
+              },
+              "type": "object"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "title": "Choice Weights"
         }
       },
       "title": "Policy",

src/typewright/cli.py

@@ -109,6 +109,8 @@ def build_parser():
     compile_p.add_argument("--seed", type=int, default=7)
     compile_p.add_argument("--max-calibration-error", type=float, default=0.05)
     compile_p.add_argument("--min-calibration-samples", type=int, default=10)
+    compile_p.add_argument("--fit-decision-knobs", action="store_true",
+                           help="Opt in to offline flat Score-cut and Choice-weight fitting.")
     backend_args(compile_p)
     teacher_args(compile_p)
     run = sub.add_parser("run", help="Run a compiled JSON artifact. Does not execute application actions.")
@@ -130,7 +132,7 @@ def build_parser():
     demo = sub.add_parser("demo", help="No-key, no-network end-to-end synthetic smoke test.")
     demo.add_argument("--out", type=Path, default=Path("runs/demo"))
     demo.add_argument("--starter", choices=["flat", "hierarchy"], default="flat")
-    inspect_p = sub.add_parser("inspect", help="Validate artifact checksum and print its typed plan.")
+    inspect_p = sub.add_parser("inspect", help="Validate artifact checksum and print its typed plan, including fitted policy knobs.")
     inspect_p.add_argument("artifact", type=Path)
     harden = sub.add_parser("harden", help="Create robustness proposals that require human label review.")
     harden.add_argument("spec", type=Path)
@@ -209,6 +211,8 @@ def dispatch(args):
             splits = {name: read_hierarchy_jsonl(getattr(args, name), draft)
                       for name in ("train", "validation", "calibration", "test")}
             validate_hierarchy_compile_inputs(source, splits)
+            if args.fit_decision_knobs:
+                raise ConfigurationError("Decision-knob fitting currently supports flat programs only.")
             options = HierarchyCompileOptions(architect=args.architect, optimizer=args.optimizer,
                 structural_rounds=args.structural_rounds, max_metric_calls=args.max_metric_calls,
                 seed=args.seed, max_calibration_error=args.max_calibration_error,
@@ -238,7 +242,7 @@ def dispatch(args):
         options = CompileOptions(architect=args.architect, optimizer=args.optimizer,
             structural_rounds=args.structural_rounds, max_metric_calls=args.max_metric_calls,
             seed=args.seed, max_calibration_error=args.max_calibration_error,
-            min_calibration_samples=args.min_calibration_samples)
+            min_calibration_samples=args.min_calibration_samples, fit_decision_knobs=args.fit_decision_knobs)
         teacher = make_teacher(args) if (args.architect == "dspy" or args.optimizer == "gepa" or args.structural_rounds) else None
         backend = make_backend(args)
         try:

src/typewright/compiler.py

@@ -23,8 +23,11 @@ class CompileOptions:
     seed: int = 7
     max_calibration_error: float = 0.05
     min_calibration_samples: int = 10
+    fit_decision_knobs: bool = False
 
     def __post_init__(self):
+        if type(self.fit_decision_knobs) is not bool:
+            raise ConfigurationError("fit_decision_knobs must be a boolean.")
         if self.architect not in {"template", "dspy"} or self.optimizer not in {"none", "gepa"}:
             raise ConfigurationError("Unknown architect or optimizer.")
         if not 0 <= self.structural_rounds <= 10:
@@ -106,7 +109,8 @@ def compile(self, source: UseCase, *, train: list[Example], validation: list[Exa
         _, calibration_predictions = evaluate(best, calibration, self.backend)
         best, calibration_fit = fit_policies(
             best, calibration, calibration_predictions,
-            max_error=options.max_calibration_error, min_samples=options.min_calibration_samples)
+            max_error=options.max_calibration_error, min_samples=options.min_calibration_samples,
+            fit_decision_knobs=options.fit_decision_knobs)
         fitted_predictions = [dict(r, decisions=apply_policy(best, r["answers"])) for r in calibration_predictions]
         calibration_report = report_from_results(best, calibration, fitted_predictions)
         best.provenance = {

src/typewright/models.py

@@ -2,9 +2,9 @@
 import math
 import re
 from pathlib import Path
-from typing import Any, Literal
+from typing import Annotated, Any, Literal
 
-from pydantic import BaseModel, ConfigDict, Field, JsonValue, model_validator
+from pydantic import BaseModel, ConfigDict, Field, JsonValue, model_serializer, model_validator
 
 from .errors import CandidateError, DataError
 from .io import atomic_json, fingerprint, load_document
@@ -107,6 +107,32 @@ class Policy(StrictModel):
     noul_threshold: float = Field(default=0.5, gt=0, lt=1)
     min_gate: float = Field(default=0.8, ge=0, le=1)
     force_review: bool = False
+    fitting_version: Literal["native-policy/v1"] | None = None
+    score_cuts: list[Annotated[float, Field(strict=True, ge=0)]] | None = None
+    choice_weights: dict[str, Annotated[float, Field(strict=True, gt=0)]] | None = None
+
+    @model_validator(mode="after")
+    def check_fitted_policy(self):
+        knobs = self.score_cuts is not None or self.choice_weights is not None
+        if knobs != (self.fitting_version is not None):
+            raise ValueError("Fitted knobs require an explicit native-policy/v1 version.")
+        if self.score_cuts is not None:
+            if not self.score_cuts or any(a >= b for a, b in zip(self.score_cuts, self.score_cuts[1:])):
+                raise ValueError("Score cuts must be nonempty and strictly ordered.")
+        if self.choice_weights is not None:
+            if not self.choice_weights or any(v <= 0 for v in self.choice_weights.values()):
+                raise ValueError("Choice weights must be positive.")
+        return self
+
+    @model_serializer(mode="wrap")
+    def serialize_compatible(self, handler):
+        # Omit absent extensions even in nested dumps: existing artifact bytes and
+        # content hashes retain their original meaning.
+        data = handler(self)
+        for key in ("fitting_version", "score_cuts", "choice_weights"):
+            if data.get(key) is None:
+                data.pop(key, None)
+        return data
 
 
 class Program(StrictModel):
@@ -132,6 +158,15 @@ def check_program(self):
         used = set()
         for name, binding in self.bindings.items():
             decision = self.decisions[name]
+            policy = self.policies[name]
+            if policy.score_cuts is not None:
+                if decision.type != "score" or len(policy.score_cuts) != len(decision.criteria) - 1:
+                    raise ValueError("Score cuts must match the declared Score scale.")
+                if any(not 0 <= cut <= len(decision.criteria) - 1 for cut in policy.score_cuts):
+                    raise ValueError("Score cuts must lie within the declared scale.")
+            if policy.choice_weights is not None:
+                if decision.type != "choice" or set(policy.choice_weights) != set(decision.criteria):
+                    raise ValueError("Choice weights must match the fixed labels.")
             if binding.kind == "question":
                 if binding.question not in self.questions:
                     raise ValueError(f"Unknown question in binding {name}.")