Repository navigation
Enforce contract rate limits (429) whenever an admission store is configured - #60
Merged
Merged
Conversation
The organization and owner rate limits Atlas stamps on the trusted envelope (UsageLimits) were only enforced when admission.enabled=true: cmd/interceptor built no admitter otherwise, and the chart ships admission.enabled=false. Per the R12 clarification (2026-10-06) there is no enablement gate for contract limits. - config: Settings.AdmissionStoreAddr (admission.valkeyAddr, else emit.valkeyAddr) and Settings.EffectiveAdmission. Store defaults (keyPrefix, leaseTtl, defaultMaxOutputTokens) are parsed regardless of admission.enabled. With admission.enabled=false the effective config clears the operator capacity tiers (contract scopes only). - interceptor: build the admitter whenever a store resolves; log loudly when no Valkey is configured at all. - proxy: admit a shared request when admission.enabled=true OR the request carries any part of the trusted envelope. Envelope-less shared requests under admission.enabled=false (historical per-resource routes) keep their previous behavior. - 429 + Retry-After stays the contract answer, 503 the operator capacity answer; operator scopes are checked first, so a tie is 503. - Tests: contract-only bursts (30 then 429) under all three QA configurations, owner contract, operator 503, precedence, R4 absent and zero, gateway and header-routed paths, real-Valkey burst. - docs/shared-tier-admission.md: what admission.enabled now means. Claude-Session: https://claude.ai/code/session_01JR59LERE7vBqUT9cga5Kvq
hhuuggoo
added a commit
that referenced
this pull request
Oct 6, 2026
phoebe #60 post-merge battery: contract-only gating, R10 reservation clamp, lint
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
QA reported (2026-10-06, "Contract rate limits are stamped on the envelope but never enforced") that an Atlas UsageLimit of 30 requests per minute reaches phoebe on the envelope (
X-Saturn-Org-Rate-Limit-Requests: 30, plus the owner, org, and auth ids), but 35-request bursts all pass in three configurations: (a) admission disabled, (b) admission enabled with no operator tiers, (c) admission enabled with an emptyorganization: {}tier.This PR makes phoebe enforce contract limits whenever an admission store (Valkey) is configured. Enforcement no longer depends on
admission.enabled, which matches the R12 clarification of 2026-10-06: contract limits are database settings and there is no enablement gate for them.Root cause
(a) is a phoebe defect.
cmd/interceptor/main.gobuildAdmissionreturned a nil admitter wheneveradmission.enabledwas false, andinternal/proxy/proxy.goonly admitted whens.admitter != nil. The chart shipsadmission.enabled: false(R5/R12 defaults), so on a default install no stamped contract limit was ever checked.(b) and (c) were not phoebe defects. The configuration QA intended was never deployed. On the uk2 sandbox (read-only inspection):
PhoebeCR has beenIrreconcilablesince 17:06Z with:admission.valkeyAddr is not a known admission key (check for a typo). The chart's unknown-key guard rejectsadmission.valkeyAddrbecause the chart takes the store address from the top-levelvalkeyAddr.v7(16:44Z), which hasadmission.enabled: false. The liveconfigmap/phoebecontains noadmission:block.checksum_configmap(9a02d814…). The rollouts at 17:07, 17:10, and 17:14 were restarts that reloaded the same configuration with admission disabled.The new tests confirm this. With admission enabled and no operator tiers, or with an empty
organization: {}tier, the existing code already returns 429 at request 31. Those subtests pass even when the old build gate is restored.Change
config: addsSettings.AdmissionStoreAddr(), which usesadmission.valkeyAddrand otherwise falls back toemit.valkeyAddr(the install Valkey; admission keys use their own prefix). Also addsSettings.EffectiveAdmission(). The key prefix, lease TTL, and default output reservation are now parsed whether or not admission is enabled. Whenadmission.enabled=false, the effective configuration clears the operator tiers, so only contract scopes are checked.cmd/interceptor: builds the admitter whenever a store resolves. When no Valkey is configured at all, it logs an ERROR that contract limits are not enforced.proxy: a shared inference request goes through admission whenadmission.enabled=true, or when it carries any part of the trusted envelope (the owner id or any scoped limit header). A shared request with no envelope at all underadmission.enabled=falsekeeps its previous behavior. Today those are only the historical per-resource routes.docs/shared-tier-admission.md: adds a table of whatadmission.enablednow controls.Tests (
internal/proxy/contract_limits_test.go; the admitter is built from loaded YAML exactly ascmd/interceptorbuilds it, and the store clock is frozen so a burst cannot cross a window boundary):TestContractOrgLimitEnforcedRegardlessOfOperatorTiers: QA cases (a), (b), (c) plus the chart-null tiers. 30×200, then 429 with "shared inference rate limit exceeded" and Retry-After between 1 and 60; the upstream is hit exactly 30 times. Uses the gateway path, so the serving mode comes from registry resolution.TestContractOwnerLimitEnforcedRegardlessOfOperatorTiers: the owner contract behaves the same way with the flag off and on.TestContractLimitEnforcedOnHeaderRoutedSharedPathWithAdmissionFlagOffTestOperatorCapacityLimitStillAnswers503TestContractAndOperatorLimitPrecedence: the tighter limit decides; an exact tie returns 503.TestAbsentContractLimitHeadersAreUnlimited(R4) andTestExplicitZeroContractLimitBlocksWithAdmissionFlagOff(R4)TestEnvelopeLessSharedRequestFollowsAdmissionFlagandTestPartialEnvelopeFailsClosedWithAdmissionFlagOffconfig:TestEffectiveAdmissionStoreAndTiers. Real Valkey:TestRealValkeyContractOnlyBurst.Red check: restoring the old build gate fails the (a) subtests, the owner, header-routed, and explicit-zero tests, and the partial-envelope test, each with 200 where 429 or 503 is expected. The (b) and (c) subtests still pass, which supports the diagnosis above. Restoring the old proxy gate (
admitter != nilalone) failsTestEnvelopeLessSharedRequestFollowsAdmissionFlag/admission.enabled=falsewith 503 where 200 is expected.Contracts
admission.enabled. An exhausted contract returns 429 + Retry-After ("shared inference rate limit exceeded"). Absent header means unlimited, and0blocks the scope (R4). A partial or malformed envelope fails closed with 503, now also whenadmission.enabled=false.admission.enabled(config key, unchanged name): it now controls only (1) the operator capacity tiers (platform,graph,organization,organizationModel,lanes) and (2) whether every shared request must carry the envelope. When it is true, a request without the envelope fails closed with 503. It is no longer an on/off switch for contract limits.admission.valkeyAddr, elseemit.valkeyAddr. With neither set, contract limits cannot be enforced and phoebe logs an ERROR at startup.admission.enabled=false, phoebe now acts on the scoped envelope headers. This relies on the edge stripping client-suppliedX-Saturn-*headers. The R8 releases already ship that strip:phoebe-inference-headersis rendered from the same trusted list, andtf-gateway-authallowlists the envelope.admission.enabled=truealready relied on the same assumption. On an edge older than that strip, a client could forge another owner's id and use up that owner's contract window. This change does not open access to anything.emit.valkeyAddr, which the admitter now falls back to. Optional follow-ups: (1) update theadmission:comment incharts/phoebe/values.yaml; it still describes the flag as the on/off switch for limits. (2) Operator tiers can still only be rendered whenadmission.enabled=true; dropping that gate would be a separate chart decision. Operations note: the uk2PhoebeCR must dropadmission.valkeyAddr(not a chart key), otherwise the release staysIrreconcilable.https://claude.ai/code/session_01JR59LERE7vBqUT9cga5Kvq