From 3c10dc1d2d66a5207111eb0963338f46624dcf69 Mon Sep 17 00:00:00 2001 From: OpenChip Framework Author <45132014+WJiangH@users.noreply.github.com> Date: Fri, 11 Sep 2026 21:04:14 -0700 Subject: [PATCH] Document difficulty-based model routing and outcome observation for orchestrator assignments OpenChip-Provenance: v1 Agent-Label: codex Agent-Role: orchestrator Agent-Provider: OpenAI Agent-Client: codex Agent-Backend: UNKNOWN Requested-Model: gpt-5.6-terra Requested-Effort: medium Observed-Identity: UNKNOWN Runtime-Attestation: none Runtime-Evidence: UNKNOWN Platform-Host: github.com Platform-Account-Login: WJiangH Platform-Account-ID: 45132014 Platform-Account-Type: User Platform-Action: commit-author-identity Platform-Evidence: https://api.github.com/users/WJiangH --- .agents/skills/orchestrator/SKILL.md | 39 +++++++++++++------ .../orchestrator/references/operating.md | 14 ++++++- docs/templates/ORCHESTRATION_STATE.md | 11 ++++++ 3 files changed, 51 insertions(+), 13 deletions(-) diff --git a/.agents/skills/orchestrator/SKILL.md b/.agents/skills/orchestrator/SKILL.md index 529a775..228c081 100644 --- a/.agents/skills/orchestrator/SKILL.md +++ b/.agents/skills/orchestrator/SKILL.md @@ -44,15 +44,18 @@ moving meanwhile. State is a record of authority, never a new authority source. for dispatch, return routing and branch retirement. Give DV independent spec-derived context; do not pass implementation-derived expected answers. Delegate through an available, authorized mechanism and record the returned durable handle. -3. Choose a configured model/effort for uncertainty, impact and context needs. - Use a strong reasoning configuration for architecture, ambiguous failures - and consequential independent review; routine mechanical work can use an - available lower-cost configuration. Respect explicit requested models; - capability or credential absence is a blocker to report, not permission to - silently substitute. Keep concrete model/provider choices in local state, - requested settings separate from observed identity, and UNKNOWN/none/UNKNOWN - where runtime identity is not independently exposed. Do not rank models from - completion counts or treat their agreement as evidence. +3. Choose a configured model/effort from the task's ambiguity/complexity, + failure impact, verification ease/cost and context needs; record those + inputs, the selected setting and rationale in the assignment. Use lighter + configured work for bounded routine tasks, and stronger reasoning for + architecture, difficult semantics, ambiguous failures and consequential + independent review. A role name alone does not choose a configuration. + Respect explicit requested models; capability or credential absence is a + blocker to report, not permission to silently substitute. Keep concrete + model/provider choices in local state, requested settings separate from + observed identity, and UNKNOWN/none/UNKNOWN where runtime identity is not + independently exposed. Do not rank models from completion counts or treat + their agreement as evidence. 4. Reuse a live assigned worker; dispatch parallel work only when dependencies, ownership and resource budgets permit it. Inspect meaningful progress and final receipts, not just process liveness. Read the substantive review @@ -60,9 +63,13 @@ moving meanwhile. State is a record of authority, never a new authority source. reruns and independent re-review, and keep the same work item/PR synchronized. Keep a PR Draft if making it Ready would trigger integration outside the recorded authorization; workflow readiness is not permission to merge. -5. Evaluate output and review quality against cited rules and raw evidence. - If results are weak, correct context, acceptance, decomposition, skill gaps - or model fit. Do not weaken gates or manufacture bugs to score workers. +5. Evaluate the complete delivery loop against cited rules and raw evidence. + Record acceptance/quality, confirmed misses or false alarms, repair cycles, + root versus human intervention, measured time/token/cost, and unknowns. + Compare only like difficulty, context, skill version and evidence scope. If + results are weak, correct context, acceptance, decomposition, skill gaps or + model fit. Do not weaken gates, manufacture bugs, or make unsupported model + ranking or maturity claims. Genuine defects and observed friction can motivate a separately owned, independently reviewed skill change; integrator curates accepted additions. After such a change, assign prospective application on the next matching @@ -76,6 +83,14 @@ moving meanwhile. State is a record of authority, never a new authority source. evidenced, or checkpoint a precise remaining dependency/authority blocker; task counts are not closure. +When configuration must change, use only a runtime-supported switch. Reconcile +the live worker, owned jobs and candidate first; preserve the original author's +ownership, a sufficient context handoff and independent review. If an in-place +switch is unavailable, record that limitation and make an explicit handoff +instead of blindly restarting or representing a requested setting as observed +identity. Each assignment also records bounded continuation or escalation +triggers. + ## Persistence and operating limits The orchestrator owns the state index; specialists own their output and receipt diff --git a/.agents/skills/orchestrator/references/operating.md b/.agents/skills/orchestrator/references/operating.md index adc31f6..c4f168a 100644 --- a/.agents/skills/orchestrator/references/operating.md +++ b/.agents/skills/orchestrator/references/operating.md @@ -13,8 +13,9 @@ handoff file for each assignment: | Contract | Work-item/requirement IDs, exact spec/profile revisions, acceptance criteria, prerequisites and unresolved questions | | Ownership | One author role, separate independent reviewer, approver and state writer; explicit transfer if replacing a worker | | Inputs and scope | Pinned source/dependency identities, allowed reads/writes, forbidden artifacts, tool/access limitations and public/private boundary | +| Model decision | Task ambiguity/complexity, failure impact, verification ease/cost and context needs; selected configured model/effort and rationale; intended independent reviewer; bounded continuation or escalation trigger | | Execution | Branch from current main (or approved dependency), worktree, requested model/effort, observed identity or UNKNOWN, commands, resource bounds and durable worker/job handles | -| Return | Changed artifacts, exact candidate, planned/selected/executed/skipped/checked scope, raw receipts and verdicts, open bugs/risks, applicable reruns, manifest PR and next action | +| Return | Changed artifacts, exact candidate, planned/selected/executed/skipped/checked scope, raw receipts and verdicts, acceptance/quality, confirmed misses or false alarms, repair cycles, root versus human intervention, measured time/token/cost and unknowns, manifest PR and next action | Freeze the independent DV context from the approved spec and golden-model sources; do not copy implementation-derived answers. Sparse worktrees or another @@ -47,6 +48,17 @@ approval. Read raw checker results and the review verdict. Missing evidence is unverified; preserve explicit executed failures according to receipt semantics. Evaluate actual friction and genuine defects, not planted bugs for agent ranking. A skill improvement is its own bounded, independently reviewed framework change. +Compare outcomes only across comparable difficulty, context, skill version and +evidence scope; they do not establish a model ranking or maturity claim. + +## Change a runtime configuration + +Use an in-place switch only when the runtime supports it. Before switching, +reconcile the live worker, all owned jobs and the candidate; retain the original +author's ownership, provide the replacement sufficient context, and preserve +independent review. If the capability is unavailable, record the limitation and +use an explicit handoff when authorized. Do not blindly restart a worker or +present a requested configuration as observed runtime identity. ## Close the work item and retire its workspace diff --git a/docs/templates/ORCHESTRATION_STATE.md b/docs/templates/ORCHESTRATION_STATE.md index ceedc75..8f6048a 100644 --- a/docs/templates/ORCHESTRATION_STATE.md +++ b/docs/templates/ORCHESTRATION_STATE.md @@ -45,6 +45,17 @@ Task status: OPEN (not started), RUNNING, WAITING (known condition), BLOCKED These are workflow states, separate from signoff acceptance. Record assignment transfer explicitly; the state index has one designated writer at a time. +For every assignment, retain task ambiguity/complexity, failure impact, +verification ease/cost and context needs; the selected configured model/effort +and rationale; intended independent reviewer; and bounded continuation or +escalation triggers. On return, retain acceptance/quality, confirmed misses or +false alarms, repair cycles, root versus human intervention, measured +time/token/cost and unknowns. Compare observations only when difficulty, +context, skill version and evidence scope are comparable; they do not establish +a model ranking or maturity claim. If changing configuration, record supported +switching capability or its limitation, reconcile the live worker/jobs/candidate, +and preserve author ownership, context handoff and independent review. + Record PR lifecycle separately: Draft / in review / changes requested / approved / merged / terminal closed, with exact head and reason. After a terminal decision, record cleanup owner, active/dependency/dirty/evidence checks, retained artifacts,