feat(ingest): board reconciler startup sweep + slow ticker + watermark (RIG-2883) - #708
Open
rigel-mintaka wants to merge 2 commits into
Open
feat(ingest): board reconciler startup sweep + slow ticker + watermark (RIG-2883)#708rigel-mintaka wants to merge 2 commits into
rigel-mintaka wants to merge 2 commits into
Conversation
…k (RIG-2883) Adds `BoardReconciler` (T3): the forge board-ingestion reliability backstop and the cold-start/backfill path, mirroring `NotifyReconciler` at repo-LIST granularity. Per sweep it enumerates enabled repos (`BoardStore.ListEnabledRepos`), conditionally lists each repo's updated-order issues since its stored watermark (`updatedLister.ListUpdatedIssues`, structurally satisfied by T2's `GitHub.ListUpdatedIssues` at T5), sinks the rows through the shared `Ingester`, then advances the durable watermark to `max(UpdatedAt)` + a fresh ETag AFTER the rows sank (advance-after-sink). A zero/absent watermark lists everything — the cold-start and App-reinstall backfill answer, no separate mechanism. `Run` fires one immediate sweep then a sweep per `Backstop` tick, returning nil on ctx cancel. Per-repo errors are isolated; `ErrBudgetExhausted` aborts the sweep, resumed next tick. Poison-row livelock is bounded: rows sink one at a time so a persistently-rejected row is skipped-and-counted while the rest sink and the watermark advances past the healthy rows (dropping the ETag so the next sweep re-lists unconditionally) — the re-walk window MUST NOT grow without bound behind a pinned watermark. `BoardStore`, `updatedLister`, and `Ingester` are the only seams; the package imports no `*store.Store` or concrete `*forge.GitHub` (wired at T5). Tests are `synctest`-driven for the time-dependent Run/ticker cases. Refs RIG-2883 Co-authored-by: Matt Wilkinson <matt@rigel.build>
|
Compass engineering docs preview: https://compass-forge-rig-2883-t3-re.compass-eng-docs.pages.dev Deployed from |
…ff note (RIG-2883 T3) Adds the multi-sweep invariant the single-sweep poison test only implied: a persistently-failing NEWEST row is retried every sweep, the re-walk window stays bounded (does not grow sweep over sweep), and the watermark floor holds at the healthy max. Also documents the deliberate tradeoff that a transient failure co-batched with a newer healthy row is dropped-until-next-update, so a future maintainer does not "fix" it back into the poison-pin livelock the isolation prevents. Refs RIG-2883 Co-authored-by: Matt Wilkinson <matt@rigel.build>
rigel-mintaka
marked this pull request as ready for review
August 28, 2026 05:41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
BoardReconciler(T3): the forge board-ingestion reliability backstop and the cold-start/backfill path, mirroringNotifyReconcilerat repo-LIST granularity.Per sweep it enumerates enabled repos (
BoardStore.ListEnabledRepos), conditionally lists each repo's updated-order issues since its stored watermark (updatedLister.ListUpdatedIssues, structurally satisfied by T2'sGitHub.ListUpdatedIssuesat T5), sinks the rows through the sharedIngester, then advances the durable watermark tomax(UpdatedAt)+ a fresh ETag AFTER the rows sank (advance-after-sink). A zero/absent watermark lists everything — the cold-start and App-reinstall backfill answer, no separate mechanism.Runfires one immediate sweep then a sweep perBackstoptick, returning nil on ctx cancel. Per-repo errors are isolated;ErrBudgetExhaustedaborts the sweep, resumed next tick.Poison-row livelock is bounded: rows sink one at a time so a persistently-rejected row is skipped-and-counted while the rest sink and the watermark advances past the healthy rows (dropping the ETag so the next sweep re-lists unconditionally) — the re-walk window MUST NOT grow without bound behind a pinned watermark.
BoardStore,updatedLister, andIngesterare the only seams; the package imports no*store.Storeor concrete*forge.GitHub(wired at T5). Tests aresynctest-driven for the time-dependent Run/ticker cases.Refs RIG-2883
Co-authored-by: Matt Wilkinson matt@rigel.build