Skip to content

Consolidate backend performance, reliability and Topology contracts - #266

Draft
n30nex wants to merge 21 commits into
devfrom
codex/release-core-20261021
Draft

n30nex wants to merge 21 commits into
devfrom
codex/release-core-20261021

Conversation

@n30nex

@n30nex n30nex commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

The backend fixes and the Topology/TRACE integration were spread across several draft branches. This candidate combines current dev with #253, #255, #259 and #256, preserving the latest observer-comparison, read-model restore, retryable timeout and Web 2.1 requirements. Collected telemetry and mobile intake are deliberately a separate dependent draft stack.

Backend performance and reliability

  • Keep upstream's bigint cursors, indexed query shapes, workload pools, read models and route/index cleanup together with bounded, paced first fills. A short work-cycle deadline no longer permanently shrinks future batches.
  • Move historical decryption out of HTTP/MQTT startup. The resumable job scans at most 500 primary-key candidates per tick, has a five-second work context, checkpoints only successful pages, and revisits history when actual channel keys change. Historical messages do not appear as new live chat.
  • Coalesce Redis failures as well as cache misses; bound cache network operations and allow canceled callers to leave shared work. Keep the late-miss fix and add bounded cache metrics.
  • Preserve missing/invalid RSSI/SNR as absent through storage, neighbor evidence and WS events, retaining valid zero. Reject oversized envelopes/topics and neighbor snapshots before expensive work.
  • Bound WS control rate, subscription/filter counts and individual writes. Surface global broadcast gaps to clients and count global/client/repeat drops.
  • Keep Topology's exact indexed time windows, shared retryable errors and TRACE quality/provenance safeguards. Add global, dense, sparse and empty-region Topology plan cases plus bounded history-page plan cases.
  • Expose optional artifact build version/revision separately from API version. Update the runtime image to supported Alpine 3.22.6 with PostgreSQL 16 client tools and UID/GID 65532, with an image smoke check.

Validation and operating boundaries

Local Go build/vet/unit suite, targeted PostgreSQL 16.15 regressions and all eight added plan cases passed. Native CI, vulnerability checks, runtime image checks and paired preview qualification now pass. The PostgreSQL fixture uses C collation. Result tests now isolate their own schema instead of relying on public tables left by another test.

No schema migration beyond upstream 001-053 is added; channel_history uses the existing background-state table. Retention and raw evidence are preserved. Operators must check writable mount ownership for the nonroot image and the documented schema-compatible rollback before deployment. Database-role separation and production soak/restore gates remain explicit operational work; this PR does not silently revoke privileges or touch live.

The paired web change adds basic Atlas and performance-enhanced Topology without a telemetry dependency. Operator documentation is in the companion docs draft. Current runtime measurements and remaining capacity gates are recorded below.

Refs #265, #252, #254, #258, #223, #224, #226. This consolidates the prior focused drafts; leave them recoverable until the candidate is accepted. No review request or production deployment is made by opening this draft.

The Topology follow-up streams adjacent array values, preserves NULL gaps and non-one array bounds, and caches the exact minute snapshot with canonical region keys. PostgreSQL result-parity, nonstandard-bound regressions and all six Topology plan cases pass; no migration is added.

Current qualification and promotion gates - 23 October

The full project roadmap, issue/PR disposition and staged promotion plan are in docs #43, tracked by docs #45. Core server #266/#267 and web #182/#183 are one paired candidate; optional server #268/#269, web #184/#185, docs #44 and Flutter #11 remain a separate rollout.

Current application heads pass native CI. Canadaverse now runs server 054e08de7d7fd581e55544e0326ca92596b9a990 and web 8dcb7f2f233c8ac4086c1ebe02f81d348630321b (2.5.1-n30nex.1), including preview extras. Its complete native frontend suite passed 1,560 tests in 172 files. Public verification matched all 34 assets and both source archives, with both MQTT brokers connected, six retained telemetry histories, and invalid-signature/unenrolled submission rejection. Preview artifacts are local builds of exact clean native-CI-qualified source; this is not a stable release.

On a restored snapshot, the committed global Topology query median improved from 1.3691 s to 0.9836 s (28.2%), with the same 9,975 edges. All six unique-traffic raster cases retained identical pixels. These are query/CPU-raster checks, not browser/GPU FPS or production throughput measurements. The later public 24h sample returned the same 9,346 links in 5.619 s cold and 0.191 s warm; earlier severe shared-host I/O pressure caused timeouts. Cold-load capacity, target-browser/device checks and production-shaped migration/recovery soak remain acceptance gates.

Production/dev/dev1 were inspected read-only. Dev1 services are stopped. This preview update changed no schema, retention, intake/worker, credentials or radio ownership. Prior restored-database qualification is recorded separately; no fresh browser or physical mobile/background qualification is claimed here. Dependency advisories and the separate Vite #186 candidate still need their own resolution and qualification. No org merge or release approval is implied.

n30nex added 20 commits October 13, 2026 04:21
Port the query casts and boundary regression from #233 (d5aacce). Keep the existing dev1 schema and API version.

@n30nex n30nex left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current consolidated head retains the upstream database changes and the focused reliability fixes, with Topology adjacency/window/cache regressions and native PostgreSQL/plan checks. This is the primary core review lane; the older focused/combined branches should be retired only after the accepted head preserves their required behavior.

The release gates are still material:

  • Live is on the three-migration baseline and still reproduces the four large-cursor failures. Dev is on schema 053. Rehearse the actual production-sized transition/rollback before a combined release; the isolated cursor hotfix can be evaluated separately.
  • Dev1's containers are stopped. Its branch name and older successful build jobs are not acceptance of this source on that host.
  • The shared Pi is currently under heavy I/O pressure. Both bounded global route-query comparisons timed out, so the faster restored-copy query result is not proof of a live-host latency improvement. Capture private workload metrics and qualify cold/warm route snapshots, ordinary API latency and ingest queues together.
  • Observer-directory traffic ranking still turns its read error into HTTP 500 in internal/api/handlers/observer_directory.go. The preview reproduced that response at about five seconds. Review retryable failures through the same 503/Retry-After contract used by the other hardened read paths; do not generalize the Topology-specific fix to every endpoint.

Keep the optional proxy/intake branch #268/#269 separate. The project-wide merge order, issue/PR dispositions and acceptance checklist are being consolidated in MeshCore-Beacon/beacon-docs#43 and tracking issue MeshCore-Beacon/beacon-docs#45. This is a comment review, not approval for production promotion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant