Skip to content

0.1.8 Beta7: Separate maintained-Provider regression status from transient availability #247

Open

Description

@NOirBRight

Parent

#319 (part of #248)

Problem

The Provider test UI currently reduces one generation POST to a binary green/error result. Current reachability, authentication, temporary capacity, protocol compatibility, and acceptance of a maintained release-regression profile are different facts.

One 502 must not erase previously accepted regression evidence, and one 200 must not create a runtime model allowlist or claim full compatibility.

Evidence states

Provider tests, Beta7 hardening, and RC evidence distinguish:

  • Verified profile — accepted versioned real-CLI regression evidence exists for that maintained route/profile.
  • Known incompatible — the requested semantic is proven unavailable on that exact route/profile.
  • Not verified — no accepted release-regression evidence; this does not prohibit a user-configured endpoint from using the generic compatibility path.
  • Temporarily unavailable — current capacity/rate/auth-pool availability without changing prior evidence.
  • Degraded — completed or recovered with a condition that cannot count as a clean release pass.

Authentication and configuration errors retain actionable subtypes and do not masquerade as Provider capacity.

Required behavior

  • Send one visible generation probe with a 30-second deadline and no hidden POST retry.
  • Offer an explicit manual Recheck action after completion.
  • Show safe route, model, protocol, time, attempt count, limited probe scope, classification, and retry/cooldown hints.
  • Treat discovery as catalog evidence only.
  • Persist no credential, prompt, or response content.
  • Never change runtime routing eligibility or choose another model from a Provider-test result.

Acceptance criteria

  • Rust/TypeScript contracts represent the evidence states and actionable auth, rate/quota, overload, timeout/connect, protocol, and model-rejection subtypes.
  • Temporary availability is not mislabeled as known incompatibility or an invalid user key.
  • Exactly one visible probe runs; Recheck is explicit.
  • A successful probe is labeled limited reachability/generation evidence, not full product support.
  • A not-verified custom endpoint remains eligible for the generic Beta2 path.
  • Retry/cooldown hints are safe and their absence is explicit.
  • Backend/UI tests cover all states, boundary timeouts, stale results, Recheck, and transitions.
  • 0.1.8 RC1: Aggregate six fixed regression batches and publish promotion evidence #257 consumes the same typed result without turning Degraded or Temporarily unavailable into a clean pass.

Blocked by

Activity

  1. added
    enhancementNew feature or request
    needs-triageMaintainer needs to evaluate this issue
    wayfinder:grillingWayfinder decision or reconciliation gate
    P1Priority 1: current iteration goal
    ready-for-agentFully specified, ready for an AFK agent
    wayfinder:taskWayfinder implementation task
    and removed
    needs-triageMaintainer needs to evaluate this issue
    wayfinder:grillingWayfinder decision or reconciliation gate
    on Jul 29, 2026
  2. changed the title [-]Separate model support from transient availability in Provider tests[/-] [+]0.1.8 Beta7: Separate maintained-Provider regression status from transient availability[/+] on Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Priority 1: current iteration goalcampaign:0.1.8-beta70.1.8 Beta7 stability and release-evidence campaignenhancementNew feature or requestready-for-agentFully specified, ready for an AFK agentwayfinder:taskWayfinder implementation task

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions