Repository navigation
Tenancy depth (quotas-only): membership-aware group quotas on the shared tier - #63
Merged
Conversation
…p_usage Membership-aware group quotas (ruled 2026-10-07, Q-T1(b)+Q-T2(b)). billing_event gains member_group_ids TEXT[] (membership evidence, never part of the money grain); the new group_usage table is the per-(group, hour) ATTRIBUTION rollup the rater writes while rating and the admission group spend check reads. Rollout: code and schema together, in the 0007 order.
…hip rides Identity The whole group scope set travels one allowlisted header (grammar frozen with the design). FromRequest reads it only through the R3 trusted-header registry (pinned fallback 13 → 14, in lockstep with the chart) and extracts the caller's group list leniently — the strict, fail-closed parse runs at the proxy, and a malformed envelope never reaches metering.
…pty) The event gains the caller's group list; the drainer binds it like every other identity column (nil for none — no empty arrays), making the column 24-wide. Recovery's duplicate-evidence compare switches to canonical JSON: Event now carries a slice and is no longer comparable with ==.
The single rate statement now writes both rollups from one snapshot: money into rated_usage (schema, grain, and write count untouched) and attribution into group_usage — each event attributed to its token's own group plus every member_group_ids group, deduped per (event, group), withheld (non-money) events contributing nothing. The same reconcile contract as rated_usage: in-window group rows the run does not reproduce are deleted. Integration harnesses apply 0008; the new conformance test pins token-group + membership + multi-group + withheld-excluded + re-rate convergence against live Postgres.
…uotas) Request gains GroupScopes: per group, the four per-minute windows run as contract scopes through the existing Lua/counter machinery and settle via the lease path (R4 sentinels end to end), and the monthly spend cap compares SUM(cost) over group_usage in Postgres — cached ~60s per (group, cap), compared in SQL so money never becomes a Go number. Spend >= cap rejects contractually (a zero cap rejects all paid work); store errors fail open with a throttled loud log (2026-09-24 posture); a scope without its group id fails closed like every broken-envelope case.
parseTrustedGroupScopes mirrors parseTrustedRateLimits: behind the admission-enabled gate, strict on the frozen grammar (v1 prefix, 32-hex gids, R4 rate sentinels, plain-decimal NUMERIC(20,9) spend caps, <=16 entries, <4 KiB) — malformed or oversize fails closed 503 with a loud log, the fail-visible default for multi-group callers. Denials map through the contractual class (429 + Retry-After); the undeclared-output reservation clamps to group generated-token limits like the org/owner ones; the parsed membership list rides the emitted event for the rater.
The monthly spend cap reads group_usage in Postgres (the rater's attribution rollup), not the admission Valkey. With a DSN the admitter gets the spend store; without one (serving-only spokes) group rate limits still enforce and the missing spend check is a loud startup log, never a silent assumption.
…elope grammar Spend check: bound each Postgres read on the request path (250ms) and pool dials (2s) so a blackholed store fails open instead of hanging requests; compute the month boundary in UTC to match the rater's UTC hour buckets; expire cached verdicts at the UTC month boundary. Envelope parser: rate fields must match the frozen grammar (reject signed and zero-padded values) and spend caps are bounded to the NUMERIC(20,9) integer width.
Cache a short-lived fail-open verdict after a spend-store error so a slow or blackholed Postgres is not re-queried once per request, collapse concurrent checks of one (group, cap) into a single in-flight read, and stop logging a client disconnect as a quota bypass, which consumed the throttled error slot a real outage needs. Add an integration-tagged test that runs the spend comparison against live Postgres loaded with the real migrations, so the zero-cap and spend-equals-cap rules cannot regress unseen.
all-unlimited group envelopes; recovery: validate member_group_ids Bound the whole group spend check by a single 250ms budget instead of one per group, so many uncached capped groups against a slow store wait at most one budget. Detach the coalesced spend read from the leader's cancellation so one client's disconnect cannot end the read every waiter shares. Under admission.enabled=false, engage the admission store only when a group scope carries a rate limit or spend cap. At the recovery boundary, reject member_group_ids lists that violate the frozen envelope contract, since recovered evidence bypasses the proxy's strict parse and the stored list is what rating attributes into group_usage.
…docs: correct the 0008 rollout and rollback claims Extend the group_usage attribution integration test so every per-event withheld gate is exercised against a shared money grain instead of vacuously, run the spend integration test under a non-UTC session timezone, pin the throttled fail-open spend log, disambiguate the group settlement tests from a kept reservation, and add a real-Valkey group scope check. Correct the 0008 rollout text (an old rater runs on the new schema and writes no group_usage) and mark the 0008 rollback lossy: dropping member_group_ids destroys the membership evidence and month-to-date group spend restarts lower.
spend path, usage-gate twins, deterministic coalescing A spend read that begins with almost none of the shared 250ms budget left and then times out was starved by earlier groups in the same request, not evidence of an outage: fail it open without the ERROR line and without a negative-cache entry so the group's next request re-queries the store. Add an end-to-end test that carries a group envelope from the proxy through drain and rating to the spend verdict, cover the usage_found and valid_usage attribution gates with unbillable twins, use grammar-valid group ids in the withheld-gates fixture, make the hung-store assertion bound against the spend budget, replace the sleep-based coalescing rendezvous with a deterministic signal, document the post-0008 drainer poison-drop against the pre-0008 schema, and correct the window_start column comment to the UTC boundary the code uses.
… reservation clamp
The membership-aware group quota build ships group RATE scopes only; the monthly spend cap machinery (member_group_ids evidence + the group_usage rollup) is deferred. No phoebe schema change ships, so 0008 and its rollout notes go; group scope enforcement is envelope-driven and safe across the cutover with no migration at all.
The monthly spend cap is deferred, so the membership-evidence pipeline it fed goes with it: metering.Event loses MemberGroupIDs, the drainer's INSERT returns to the 23-column shape, identity stops extracting the group list (Identity keeps carrying the raw envelope to the proxy — the R3 gate and the strict proxy parse are the quota path), and recovery's duplicate comparison returns to == now that Event carries no slice.
The frozen 5-field grammar stays — v1;<gid>:<req>,<tot>,<unc>,<gen>,<spend> — but the fifth field is now the parked-activation boundary for the deferred monthly spend cap: it MUST be empty, and any non-empty value, well-formed decimal included, is malformed and fails closed (503) with a loud log at the parse. Activating spend caps later is a parser change at this one site, not an envelope renegotiation. anyGroupScopeEnforced and the admission-trigger comments lose the spend clause; group rate scopes, the fail-closed paths, and the undeclared-max_tokens reservation clamp are untouched. The emit() member_group_ids hunk reverts with the metering commit's revert.
The group attribution rollup, its reconcile, and the Result counters existed only to feed the deferred monthly spend cap, so the rater returns to rated_usage-only behavior: money rows are written exactly once, at the unchanged grain, in one statement. The group rollup integration tests and the 0008 rollback-loss test go with the migration; the e2e harness no longer applies 0008, and the group-spend pipeline test (the only e2e consumer of the admission spend store) goes with it.
buildAdmission returns the admitter without a spend store: the DATABASE_URL switch, the PostgresSpendStore construction, the enabled/DISABLED/NOT-enforced startup logs, and the spend-wiring tests (including the integration test, whose whole subject was the wiring) go with the deferred monthly spend cap. Group rate limits enforce from the envelope with no database at all, so a serving-only spoke install is unchanged.
The monthly spend cap machinery — PostgresSpendStore, the verdict cache, the singleflight, the throttled failure log, and the Admit pre-check — is deferred, so spend.go and its integration test are gone and internal/admission/admission.go returns to rate scopes only: GroupScope keeps GroupID + RateLimits, the admitter loses the spend-store fields and WithGroupSpend, and the anonymous-group fail-closed check keys on limits alone. The group rate scopes and the Lua/counter machinery are untouched; the group rate tests keep their miniredis doubles with the spend-store helper dropped (the cap locals the revive rename had called spendCap are rate limits again, so they read reqCap/genCap).
TestParseTrustedGroupScopes/over_the_size_bound feeds a header over maxGroupScopeBytes and asserts the error names the 4096-byte envelope bound, so deleting or weakening the size check in admission.go goes red.
TestGroupTotalPromptTokensWindowEnforced caps only TotalPromptTokens on a group scope, settles each 10-token input estimate, and asserts the next Admit is a Contractual rejection scoped group:<gid> — so group wiring that drops a prompt window for groups only can no longer stay green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Tenancy depth, quotas-only re-cut per Hugo's Q-T3 ruling (rulings.md 2026-10-07 row 173: "ok. lets go with shard tier caps and we can ignore the monthly budget for now"): membership-aware group usage quotas (the four per-minute rates) enforced at admission on the shared-gateway path only. The monthly-spend-cap half of the original Q-T1(b) is deferred — parked, retrievable later. Supersedes #62 (the full quotas+spend version; see closing comment).
X-Saturn-Group-Scopes(R3 lockstep with the saturn-k8s chart default in #1085). Frozen grammarv1;<gid>:<req>,<tot>,<unc>,<gen>,<spend>;<gid>:...— the spend field is reserved-empty this build; a non-empty spend field is malformed → fail-closed 503 + loud log, nothing metered (the parked-activation boundary, pinned byTestSharedGroupScopesNonEmptySpendFailsClosed).Test plan
go build/vet(both tag sets), full unit suite green incl. group envelope parse (valid/multi/malformed/oversize/non-empty-spend), rate scopes (requests/prompt/generated windows), clamp, fail-open paths; integration-tagged suites compile and run undermake integration-testwith a live Postgres.Battery
Tier 2 (trust boundary + admission), 2 rounds: ESCALATE-unsigned (protocol: Hugo's snapshot sign-off at merge). Review-clean; 2 test-pinning fixes (
94cfd26size-bound test,2c503b3total-prompt window test). Re-cut fidelity: every remaining hunk is a spend excision; quota machinery byte-identical to the battery-reviewed branch. Report:agent-wip/battery-reports/battery-tenancy-quotas-phoebe.md.Contracts
Pairs with the saturn quotas PR (stamper) and #1085 (chart surfaces). Runtime-insensitive to order; absence of the header = unlimited (R4). Follow-ups (not blocking): Q-T4 authoring surface (RULED: org admins tweak user/group limits in their org — implementation lands as a follow-up PR), Q-T6 chart/phoebe lockstep guard.