Skip to content

Enforce contract rate limits (429) whenever an admission store is configured - #60

Merged
hhuuggoo merged 1 commit into
release-2026.08.01from
hugo/contract-limits-enforced
Oct 6, 2026
Merged

hhuuggoo merged 1 commit into
release-2026.08.01from
hugo/contract-limits-enforced

Conversation

@hhuuggoo

@hhuuggoo hhuuggoo commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Summary

QA reported (2026-10-06, "Contract rate limits are stamped on the envelope but never enforced") that an Atlas UsageLimit of 30 requests per minute reaches phoebe on the envelope (X-Saturn-Org-Rate-Limit-Requests: 30, plus the owner, org, and auth ids), but 35-request bursts all pass in three configurations: (a) admission disabled, (b) admission enabled with no operator tiers, (c) admission enabled with an empty organization: {} tier.

This PR makes phoebe enforce contract limits whenever an admission store (Valkey) is configured. Enforcement no longer depends on admission.enabled, which matches the R12 clarification of 2026-10-06: contract limits are database settings and there is no enablement gate for them.

Root cause

(a) is a phoebe defect. cmd/interceptor/main.go buildAdmission returned a nil admitter whenever admission.enabled was false, and internal/proxy/proxy.go only admitted when s.admitter != nil. The chart ships admission.enabled: false (R5/R12 defaults), so on a default install no stamped contract limit was ever checked.

(b) and (c) were not phoebe defects. The configuration QA intended was never deployed. On the uk2 sandbox (read-only inspection):

  • The Phoebe CR has been Irreconcilable since 17:06Z with: admission.valkeyAddr is not a known admission key (check for a typo). The chart's unknown-key guard rejects admission.valkeyAddr because the chart takes the store address from the top-level valkeyAddr.
  • So the last successful release is helm v7 (16:44Z), which has admission.enabled: false. The live configmap/phoebe contains no admission: block.
  • Every interceptor revision from 16:44Z to 17:14Z has the same checksum_configmap (9a02d814…). The rollouts at 17:07, 17:10, and 17:14 were restarts that reloaded the same configuration with admission disabled.
  • So all three bursts actually ran as case (a).

The new tests confirm this. With admission enabled and no operator tiers, or with an empty organization: {} tier, the existing code already returns 429 at request 31. Those subtests pass even when the old build gate is restored.

Change

  • config: adds Settings.AdmissionStoreAddr(), which uses admission.valkeyAddr and otherwise falls back to emit.valkeyAddr (the install Valkey; admission keys use their own prefix). Also adds Settings.EffectiveAdmission(). The key prefix, lease TTL, and default output reservation are now parsed whether or not admission is enabled. When admission.enabled=false, the effective configuration clears the operator tiers, so only contract scopes are checked.
  • cmd/interceptor: builds the admitter whenever a store resolves. When no Valkey is configured at all, it logs an ERROR that contract limits are not enforced.
  • proxy: a shared inference request goes through admission when admission.enabled=true, or when it carries any part of the trusted envelope (the owner id or any scoped limit header). A shared request with no envelope at all under admission.enabled=false keeps its previous behavior. Today those are only the historical per-resource routes.
  • docs/shared-tier-admission.md: adds a table of what admission.enabled now controls.

Tests (internal/proxy/contract_limits_test.go; the admitter is built from loaded YAML exactly as cmd/interceptor builds it, and the store clock is frozen so a burst cannot cross a window boundary):

  • TestContractOrgLimitEnforcedRegardlessOfOperatorTiers: QA cases (a), (b), (c) plus the chart-null tiers. 30×200, then 429 with "shared inference rate limit exceeded" and Retry-After between 1 and 60; the upstream is hit exactly 30 times. Uses the gateway path, so the serving mode comes from registry resolution.
  • TestContractOwnerLimitEnforcedRegardlessOfOperatorTiers: the owner contract behaves the same way with the flag off and on.
  • TestContractLimitEnforcedOnHeaderRoutedSharedPathWithAdmissionFlagOff
  • TestOperatorCapacityLimitStillAnswers503
  • TestContractAndOperatorLimitPrecedence: the tighter limit decides; an exact tie returns 503.
  • TestAbsentContractLimitHeadersAreUnlimited (R4) and TestExplicitZeroContractLimitBlocksWithAdmissionFlagOff (R4)
  • TestEnvelopeLessSharedRequestFollowsAdmissionFlag and TestPartialEnvelopeFailsClosedWithAdmissionFlagOff
  • config: TestEffectiveAdmissionStoreAndTiers. Real Valkey: TestRealValkeyContractOnlyBurst.

Red check: restoring the old build gate fails the (a) subtests, the owner, header-routed, and explicit-zero tests, and the partial-envelope test, each with 200 where 429 or 503 is expected. The (b) and (c) subtests still pass, which supports the diagnosis above. Restoring the old proxy gate (admitter != nil alone) fails TestEnvelopeLessSharedRequestFollowsAdmissionFlag/admission.enabled=false with 503 where 200 is expected.

Contracts

  • Contract limits (Atlas UsageLimits on the trusted envelope) are now enforced whenever the envelope carries them, independent of operator tiers and of admission.enabled. An exhausted contract returns 429 + Retry-After ("shared inference rate limit exceeded"). Absent header means unlimited, and 0 blocks the scope (R4). A partial or malformed envelope fails closed with 503, now also when admission.enabled=false.
  • 503 + Retry-After stays the answer for operator capacity scopes. Admission-store unavailability is unchanged: the soft gate is bypassed and the bypass is logged. Precedence is pinned: operator scopes are checked before contract scopes in one transaction, so when both are exhausted at the same request the answer is 503.
  • Meaning of admission.enabled (config key, unchanged name): it now controls only (1) the operator capacity tiers (platform, graph, organization, organizationModel, lanes) and (2) whether every shared request must carry the envelope. When it is true, a request without the envelope fails closed with 503. It is no longer an on/off switch for contract limits.
  • Admission store resolution: admission.valkeyAddr, else emit.valkeyAddr. With neither set, contract limits cannot be enforced and phoebe logs an ERROR at startup.
  • Trust assumption (security note): under admission.enabled=false, phoebe now acts on the scoped envelope headers. This relies on the edge stripping client-supplied X-Saturn-* headers. The R8 releases already ship that strip: phoebe-inference-headers is rendered from the same trusted list, and tf-gateway-auth allowlists the envelope. admission.enabled=true already relied on the same assumption. On an edge older than that strip, a client could forge another owner's id and use up that owner's contract window. This change does not open access to anything.
  • Operational: on a default install, every shared gateway inference request now makes one admission Valkey call (Lua script, 100 ms per-operation budget, fails open).
  • saturn-k8s: no default change is required for enforcement. The chart always renders emit.valkeyAddr, which the admitter now falls back to. Optional follow-ups: (1) update the admission: comment in charts/phoebe/values.yaml; it still describes the flag as the on/off switch for limits. (2) Operator tiers can still only be rendered when admission.enabled=true; dropping that gate would be a separate chart decision. Operations note: the uk2 Phoebe CR must drop admission.valkeyAddr (not a chart key), otherwise the release stays Irreconcilable.

https://claude.ai/code/session_01JR59LERE7vBqUT9cga5Kvq

The organization and owner rate limits Atlas stamps on the trusted
envelope (UsageLimits) were only enforced when admission.enabled=true:
cmd/interceptor built no admitter otherwise, and the chart ships
admission.enabled=false. Per the R12 clarification (2026-10-06) there is
no enablement gate for contract limits.

- config: Settings.AdmissionStoreAddr (admission.valkeyAddr, else
  emit.valkeyAddr) and Settings.EffectiveAdmission. Store defaults
  (keyPrefix, leaseTtl, defaultMaxOutputTokens) are parsed regardless
  of admission.enabled. With admission.enabled=false the effective
  config clears the operator capacity tiers (contract scopes only).
- interceptor: build the admitter whenever a store resolves; log loudly
  when no Valkey is configured at all.
- proxy: admit a shared request when admission.enabled=true OR the
  request carries any part of the trusted envelope. Envelope-less
  shared requests under admission.enabled=false (historical
  per-resource routes) keep their previous behavior.
- 429 + Retry-After stays the contract answer, 503 the operator
  capacity answer; operator scopes are checked first, so a tie is 503.
- Tests: contract-only bursts (30 then 429) under all three QA
  configurations, owner contract, operator 503, precedence, R4 absent
  and zero, gateway and header-routed paths, real-Valkey burst.
- docs/shared-tier-admission.md: what admission.enabled now means.

Claude-Session: https://claude.ai/code/session_01JR59LERE7vBqUT9cga5Kvq
@hhuuggoo
hhuuggoo merged commit b2ee1ab into release-2026.08.01 Oct 6, 2026
4 checks passed
hhuuggoo added a commit that referenced this pull request Oct 6, 2026
phoebe #60 post-merge battery: contract-only gating, R10 reservation clamp, lint
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant