Skip to content

Run index: list runs from small summaries, sweep without stalling the gateway - #60

Merged
pallaoro merged 1 commit into
mainfrom
run-index
Sep 25, 2026
Merged

pallaoro merged 1 commit into
mainfrom
run-index

Conversation

@pallaoro

Copy link
Copy Markdown
Member

listRuns parsed every stored record to build one page, so its cost grew with
the bytes stored rather than the number of runs; with megabyte-sized records a
3-run page took seconds. The daily retention sweep did the same inside the
gateway process, synchronously.

  • The store keeps a small summary of each top-level run at
    stateDir/index/.json, written after every record write. An entry names
    the record version it was built from (mtime to the millisecond, and size) and
    is used only while the record still matches, so a record rewritten by any
    process is never listed stale. The index is derived data: a missing or stale
    entry means the record is parsed. Older readers skip it (the name starts
    with "
    ").
  • listRuns reads entries and parses a record only when its entry is missing or
    stale. Measured on 150 runs / 273 MB: 243 ms per page before, 3 ms after.
  • sweepRuns replaces pruneRuns: it backfills missing or stale entries, applies
    retention using the entries, drops entries whose run is gone, and yields to
    the event loop as it goes. It runs daily in the serving process whether or
    not retention is enabled.
  • Version 1.7.0.

… gateway

listRuns parsed every stored record to build one page, so its cost grew with
the bytes stored rather than the number of runs; with megabyte-sized records a
3-run page took seconds. The daily retention sweep did the same inside the
gateway process, synchronously.

- The store keeps a small summary of each top-level run at
  stateDir/_index/<run>.json, written after every record write. An entry names
  the record version it was built from (mtime to the millisecond, and size) and
  is used only while the record still matches, so a record rewritten by any
  process is never listed stale. The index is derived data: a missing or stale
  entry means the record is parsed. Older readers skip it (the name starts
  with "_").
- listRuns reads entries and parses a record only when its entry is missing or
  stale. Measured on 150 runs / 273 MB: 243 ms per page before, 3 ms after.
- sweepRuns replaces pruneRuns: it backfills missing or stale entries, applies
  retention using the entries, drops entries whose run is gone, and yields to
  the event loop as it goes. It runs daily in the serving process whether or
  not retention is enabled.
- Version 1.7.0.
@pallaoro
pallaoro merged commit cd00860 into main Sep 25, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant