Repository navigation
Tenancy depth: membership-aware group quotas + monthly group spend caps - #62
Closed
hhuuggoo wants to merge 17 commits into
Conversation
…p_usage Membership-aware group quotas (ruled 2026-10-07, Q-T1(b)+Q-T2(b)). billing_event gains member_group_ids TEXT[] (membership evidence, never part of the money grain); the new group_usage table is the per-(group, hour) ATTRIBUTION rollup the rater writes while rating and the admission group spend check reads. Rollout: code and schema together, in the 0007 order.
…hip rides Identity The whole group scope set travels one allowlisted header (grammar frozen with the design). FromRequest reads it only through the R3 trusted-header registry (pinned fallback 13 → 14, in lockstep with the chart) and extracts the caller's group list leniently — the strict, fail-closed parse runs at the proxy, and a malformed envelope never reaches metering.
…pty) The event gains the caller's group list; the drainer binds it like every other identity column (nil for none — no empty arrays), making the column 24-wide. Recovery's duplicate-evidence compare switches to canonical JSON: Event now carries a slice and is no longer comparable with ==.
The single rate statement now writes both rollups from one snapshot: money into rated_usage (schema, grain, and write count untouched) and attribution into group_usage — each event attributed to its token's own group plus every member_group_ids group, deduped per (event, group), withheld (non-money) events contributing nothing. The same reconcile contract as rated_usage: in-window group rows the run does not reproduce are deleted. Integration harnesses apply 0008; the new conformance test pins token-group + membership + multi-group + withheld-excluded + re-rate convergence against live Postgres.
…uotas) Request gains GroupScopes: per group, the four per-minute windows run as contract scopes through the existing Lua/counter machinery and settle via the lease path (R4 sentinels end to end), and the monthly spend cap compares SUM(cost) over group_usage in Postgres — cached ~60s per (group, cap), compared in SQL so money never becomes a Go number. Spend >= cap rejects contractually (a zero cap rejects all paid work); store errors fail open with a throttled loud log (2026-09-24 posture); a scope without its group id fails closed like every broken-envelope case.
parseTrustedGroupScopes mirrors parseTrustedRateLimits: behind the admission-enabled gate, strict on the frozen grammar (v1 prefix, 32-hex gids, R4 rate sentinels, plain-decimal NUMERIC(20,9) spend caps, <=16 entries, <4 KiB) — malformed or oversize fails closed 503 with a loud log, the fail-visible default for multi-group callers. Denials map through the contractual class (429 + Retry-After); the undeclared-output reservation clamps to group generated-token limits like the org/owner ones; the parsed membership list rides the emitted event for the rater.
The monthly spend cap reads group_usage in Postgres (the rater's attribution rollup), not the admission Valkey. With a DSN the admitter gets the spend store; without one (serving-only spokes) group rate limits still enforce and the missing spend check is a loud startup log, never a silent assumption.
…elope grammar Spend check: bound each Postgres read on the request path (250ms) and pool dials (2s) so a blackholed store fails open instead of hanging requests; compute the month boundary in UTC to match the rater's UTC hour buckets; expire cached verdicts at the UTC month boundary. Envelope parser: rate fields must match the frozen grammar (reject signed and zero-padded values) and spend caps are bounded to the NUMERIC(20,9) integer width.
Cache a short-lived fail-open verdict after a spend-store error so a slow or blackholed Postgres is not re-queried once per request, collapse concurrent checks of one (group, cap) into a single in-flight read, and stop logging a client disconnect as a quota bypass, which consumed the throttled error slot a real outage needs. Add an integration-tagged test that runs the spend comparison against live Postgres loaded with the real migrations, so the zero-cap and spend-equals-cap rules cannot regress unseen.
all-unlimited group envelopes; recovery: validate member_group_ids Bound the whole group spend check by a single 250ms budget instead of one per group, so many uncached capped groups against a slow store wait at most one budget. Detach the coalesced spend read from the leader's cancellation so one client's disconnect cannot end the read every waiter shares. Under admission.enabled=false, engage the admission store only when a group scope carries a rate limit or spend cap. At the recovery boundary, reject member_group_ids lists that violate the frozen envelope contract, since recovered evidence bypasses the proxy's strict parse and the stored list is what rating attributes into group_usage.
…docs: correct the 0008 rollout and rollback claims Extend the group_usage attribution integration test so every per-event withheld gate is exercised against a shared money grain instead of vacuously, run the spend integration test under a non-UTC session timezone, pin the throttled fail-open spend log, disambiguate the group settlement tests from a kept reservation, and add a real-Valkey group scope check. Correct the 0008 rollout text (an old rater runs on the new schema and writes no group_usage) and mark the 0008 rollback lossy: dropping member_group_ids destroys the membership evidence and month-to-date group spend restarts lower.
spend path, usage-gate twins, deterministic coalescing A spend read that begins with almost none of the shared 250ms budget left and then times out was starved by earlier groups in the same request, not evidence of an outage: fail it open without the ERROR line and without a negative-cache entry so the group's next request re-queries the store. Add an end-to-end test that carries a group envelope from the proxy through drain and rating to the spend verdict, cover the usage_found and valid_usage attribution gates with unbillable twins, use grammar-valid group ids in the withheld-gates fixture, make the hung-store assertion bound against the spend budget, replace the sleep-based coalescing rendezvous with a deterministic signal, document the post-0008 drainer poison-drop against the pre-0008 schema, and correct the window_start column comment to the UTC boundary the code uses.
… reservation clamp
Contributor
Author
|
Superseded by #63. Hugo's Q-T3 ruling (2026-10-07, rulings.md row 173) re-cut this build to shared-tier group QUOTAS only — monthly spend caps are deferred ('ignore for now', retrievable). This branch's full implementation (quotas + spend caps, Battery Tier 2-reviewed) stays parked at head edada95 for retrieval when spend caps come back. The quotas-only re-cut (phoebe branch ctask/tenancy-quotas-…, fresh Battery Tier 2 clean) is #63. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Tenancy depth (WHITEPAPER-AUDIT row 31): membership-aware group quotas (the four per-minute rates) and group monthly spend caps on the shared gateway path, per Hugo's rulings Q-T1=(b) and Q-T2=(b) (rulings.md, 2026-10-07). The org level and the billing grain are unchanged; money is still written exactly once per window.
billing_event.member_group_ids(evidence, NULL when empty) +group_usageATTRIBUTION rollupPK(group_id, window_start)— not money; same reconcile contract asrated_usage.X-Saturn-Group-Scopesjoins the pinned trusted set (13→14, R3 lockstep with the saturn-k8s chart default). Frozen grammarv1;<gid>:<req>,<tot>,<unc>,<gen>,<spend>;...— empty=unlimited,0=zero cap, plain-decimal spend ≤9 fraction digits, ≤16 entries, <4KiB; malformed/oversize → fail-closed 503 + loud log; strict parse lives at the proxy so a malformed envelope never reaches metering.billing_event; the rater upsertsgroup_usageper (group, hour) from tokengroup_id∪member_group_idswhile writing the single money row; withheld events excluded.SUM(group_usage.cost)for the calendar month, cached ~60s per (group, cap), errors not cached so it self-heals, fail-open + throttled loud log per the 2026-09-24 posture). Cap reached → 429 + Retry-After (contractual class). Multi-group is fail-visible: usage counts against EVERY limited group.Test plan
-raceproxy + waker) green.-tags=integration, PHOEBE_TEST_DATABASE_URL) green against Postgres 16 with real 0008 DDL, incl. group-attribution and buildAdmission spend-store wiring tests.Battery
Tier 2 (trust-boundary + money + schema), 4 rounds: ESCALATE-unsigned (protocol: Tier 2 needs Hugo's snapshot sign-off at merge). 32 findings fix-verified, 2 refuted with executed proof, gate green on the final snapshot. Two design escalations filed as Q-T8 (duplicate header lines → fail closed?) and Q-T9 (startup-DB-outage spend-cap semantics); both are ruling questions, not defects. One bounded low residual: slow-store starvation floor (evidence in the battery report). Full report:
agent-wip/battery-reports/battery-tenancy-phoebe.mdin saturn-architecture-docs.Contracts / deploy order