Repository navigation
feat: Add Enterprise 1000-user scaling, SSE quota accounting, and bulk provisioning - #33
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
|
||
| tmpdir = tempfile.mkdtemp(prefix="ent1k_broker_1000_") | ||
| db_path = os.path.join(tmpdir, "broker_1000.db") | ||
| admin_key = "adm-broker-1000-secret" |
There was a problem hiding this comment.
🛑 Gitleaks has detected a secret with rule-id generic-api-key in commit 464b314.
If this secret is a true positive, please rotate the secret ASAP.
If this secret is a false positive, you can add the fingerprint below to your .gitleaksignore file and commit the change to this branch.
echo 464b314e742db81e4a206d15416ca3f789361295:tests/test_scale_1000_users.py:generic-api-key:389 >> .gitleaksignore
There was a problem hiding this comment.
Fixed in 5506f65: replaced test token literal in tests/test_scale_1000_users.py and amended commit so Gitleaks scans clean.
✅ Security Check Results
Все проверки пройдены. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 464b314e74
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| print(f" Max body: {MAX_BODY_BYTES//1024}KB", flush=True) | ||
|
|
||
| server = HTTPServer(("0.0.0.0", PORT), Handler) | ||
| server = ThreadingHTTPServer(("0.0.0.0", PORT), Handler) |
There was a problem hiding this comment.
With the threaded server, simultaneous requests for the same instance can all pass _check_quota() before any upstream response records usage. The database locks protect individual operations, not the admission/accounting interval, so the quota no longer bounds concurrent spending. Using the existing mock upstream with a barrier, eight requests against a 30-token limit all returned 200 and accumulated 240 tokens; the previous serial server admitted only the first. Add per-instance admission synchronization or outstanding-token reservations, and account for shared department limits as well.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 5506f65: added per-instance and per-department admission locks (_admission_guard) across the check-to-accounting window in quota-proxy/proxy.py, plus regression test test_concurrent_same_instance_quota_admission_barrier in tests/test_quota_enterprise.py.
| broker_key = secrets.token_urlsafe(32) | ||
| quota_key = f"quota-{spec.name}-" + secrets.token_urlsafe(24) | ||
| gateway_token = secrets.token_urlsafe(32) | ||
| name_upper = spec.name.upper().replace("-", "_") |
There was a problem hiding this comment.
Importing the valid, distinct names user-1 and user_1 succeeds, but both normalize to USER_1. The importer consequently writes duplicate broker/quota variables, and both generated services reference the same variables. Once the environment is loaded, one user's credentials overwrite the other's, collapsing their broker identity and quota accounting. Validate normalized names against both the batch and existing fleet before creating files, or use an unambiguous encoding.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 5506f65: added normalized ENV_SUFFIX collision detection across both the import batch and existing fleet in scripts/bulk-users.py and scripts/add-user.sh, covered by unit tests in tests/test_bulk_provisioning.py.
| name = k[len("BROKER_KEY_"):].lower() | ||
| keys[hashlib.sha256(v.strip().encode()).hexdigest()] = name | ||
| return keys | ||
| name = k[len("BROKER_KEY_"):].lower().strip().replace("_", "-") |
There was a problem hiding this comment.
For an existing supported instance named team_one, BROKER_KEY_TEAM_ONE previously authenticated as team_one, but this replacement now authenticates it as team-one. Requests addressed to team_one consequently return Unknown recipient, and the original inbox is no longer accessible through that key. Both provisioning scripts still accept underscores, so this affects valid instances even without a naming collision. Preserve the original identity through explicit metadata or provide a compatible migration rather than unconditionally converting underscores.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 5506f65: preserved underscore-containing instance identities in broker/broker.py via _resolve_env_instance_name() and alias resolution (_instance_aliases), covered by test_underscore_and_hyphen_env_identities_preserved in tests/test_broker_enterprise.py.
| env_lines_to_append.extend( | ||
| [ | ||
| f"BROKER_KEY_{name_upper}={broker_key}", | ||
| f"QUOTA_KEY_{name_upper}={quota_key}", | ||
| f"QUOTA_LIMIT_{name_upper}={spec.quota_limit}", | ||
| ] |
There was a problem hiding this comment.
When a manifest assigns a user to engineering, the importer stores that department only in an instance comment and Compose label; the root environment entries omit QUOTA_DEPT_<NAME>, and the importer never registers the department through the quota API. The proxy reads department membership only from those environment variables or quota_keys, so even after recreating the proxy, imported users belong to default and bypass any configured engineering quota. Persist the department alongside the generated quota key or register it explicitly.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 5506f65: persisted QUOTA_DEPT_<ENV_SUFFIX>=<department> to the root .env in scripts/bulk-users.py and verified in tests/test_bulk_provisioning.py.
…k provisioning - quota-proxy: add SSEUsageParser for text/event-stream token accounting (Anthropic & OpenAI) with 64KB line cap, dynamic key registration (/quota/register-key, /quota/reload-keys), department limits (/quota/set-department-limit), paginated /quota/report, ThreadingHTTPServer, and SQLite WAL tuning - broker: add dynamic key registration (/admin/register-key, /admin/reload-keys), message priorities, custom TTL, admin /broadcast, paginated /inbox (limit, offset, since_id, peek) and /inbox/all, and throttled expiry purge - fleet & ops: add scripts/bulk-users.py (import, export, status) with Compose shard profiles and isolated shard networks, batch self-healing in scripts/sync-instances.sh, per-caller rate limits in service-agent, and parallel container stats polling in resource-monitor - benchmark & tests: add scripts/benchmark-1000-users.py, docs/ENTERPRISE_1000_USERS.md, and comprehensive enterprise & 1000-user scale test suites (164 total tests passing)
464b314 to
5506f65
Compare
🤖 I have created a release *beep* *boop* --- ## [1.0.0](v0.6.0...v1.0.0) (2026-10-26) ### ⚠ BREAKING CHANGES * Migrate to Hermes Agent (v2.0.0) ([#31](#31)) ### Features * Add Enterprise 1000-user scaling, SSE quota accounting, and bulk provisioning ([#33](#33)) ([02997fb](02997fb)) * Migrate to Hermes Agent (v2.0.0) ([#31](#31)) ([c68d20e](c68d20e)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Summary
Transforms CortexForge v2.0.0 on Hermes Agent into an Enterprise Production-Ready platform capable of onboarding, operating, monitoring, and enforcing quotas across 1,000 concurrent users.
Type of Change
Changes
quota-proxy/proxy.py):SSEUsageParser) fortext/event-streamresponses (Anthropicmessage_start/message_deltaand OpenAIusagechunks) with 64 KB line-buffer protection against OOM.POST /quota/register-key,POST /quota/reload-keys) backed by SQLitequota_keys.POST /quota/set-department-limit) and pagination/filtering onGET /quota/report(department,status,limit,offset).ThreadingHTTPServerwith SQLite WAL mode (busy_timeout=10000,synchronous=NORMAL) and boundedRateLimiterbucket eviction.broker/broker.py):POST /admin/register-key,POST /admin/reload-keys), message priorities (low/normal/high/urgent), customttl_sec, andPOST /broadcast(including"*"for all registered agents).peek=1onGET /inboxandGET /inbox/all, plus throttled background TTL expiry purging.scripts/bulk-users.py,scripts/sync-instances.sh,service-agent/server.py,resource-monitor/monitor.py):scripts/bulk-users.py(import,export,status) with Docker Compose shard profiles (shard-0..shard-9), isolated shard bridge networks (corp-shard-<N>to avoid Linux bridge 1024-port exhaustion), atomic.envupdates, and symlink protection.scripts/sync-instances.shto inspect all containers in a singledocker ps -acall and restart stopped containers in batches of 50.SKILL_RATE_LIMIT) and configurable concurrency (MAX_CONCURRENT_SKILLS) inservice-agent/server.py, and parallel container stats polling (ThreadPoolExecutor(max_workers=32)) + pagination inresource-monitor/monitor.py.scripts/benchmark-1000-users.py,docs/ENTERPRISE_1000_USERS.md,tests/):scripts/benchmark-1000-users.py(make benchmark-1k) and 32 new enterprise/scale unit tests (164/164total tests passing; 5,020 benchmark ops across 1,000 concurrent users at ~355 RPS with 0 errors).Testing
python3 -m unittest discover tests/ -v— 164/164 tests passpython3 scripts/benchmark-1000-users.py— 5,020 ops across 1,000 users, 0 errorspython3 .github/scripts/ai_security_check.py— no findingsshellcheck scripts/*.sh— 0 warnings / 0 errorsDocumentation
README.md,README.ru.md,CLAUDE.md)docs/updated (docs/ENTERPRISE_1000_USERS.md)CHANGELOG.mdentry added (or release-please will handle it)Security
.env.examplesubprocess.run()calls without input validation