Skip to content

store: paginate catalogue sync, enable progressive UI population, and add 1-hour cache - #1022

Open
mubashardev wants to merge 6 commits into
JingMatrix:masterfrom
mubashardev:feat/store-paginated-sync
Open

mubashardev wants to merge 6 commits into
JingMatrix:masterfrom
mubashardev:feat/store-paginated-sync

Conversation

@mubashardev

@mubashardev mubashardev commented Oct 20, 2026 •

Copy link
Copy Markdown

Background and Endpoint Status

The current store implementation queries the monolithic catalogue at:
https://modules.lsposed.org/modules.json

This endpoint now returns HTTP 403 Forbidden. As a result, the client permanently falls back to:
https://backup.modules.lsposed.org/modules.json

However, this backup host serves an archived, static snapshot of the catalogue. It does not reflect new module submissions or new module versions.

The live LSPosed repository infrastructure exposes two active endpoints:

  1. Paginated Catalogue Index:
    GET https://modules.lsposed.org/module-list/{page}.json
    Serves the complete module catalogue in chunks of ~30 modules per page (currently 40 pages, 1,180+ active modules), including fields for page, pageCount, and total.
  2. Module Detail and Releases:
    GET https://modules.lsposed.org/module/{packageName}.json
    Serves full release notes, README HTML, and historical releases for an individual module.

Technical Changes

1. Endpoint Architecture and Mirror Routing (RepoRepository.kt)

  • Primary Catalogue Endpoint: Replaced the blocked monolithic URL with sequential calls to https://modules.lsposed.org/module-list/{page}.json.
  • Detail Mirror Precedence: Updated DETAIL_MIRRORS so https://modules.lsposed.org/ takes precedence over https://backup.modules.lsposed.org/, ensuring release changelogs and download assets are served from the primary host.
  • Fallbacks: Retained https://backup.modules.lsposed.org/modules.json as an offline/legacy fallback if page 1 network requests fail.

2. Sequential Background Sync Pipeline

  • Initial Page Fast Path: Fetches page 1 synchronously on the IO dispatcher so the first ~30 modules render immediately when opening the Store tab without waiting for the full repository.
  • Non-blocking Sequential Ingestion: Dispatches a sequential loop for pages 2 through pageCount in the background. Sequenced requests prevent mirror rate-limiting and ensure deterministic catalog accumulation.
  • Stream Parsing and Memory Efficiency: Parses the paginated JSON body through JsonParser.parseReader into structured OnlineModule entries, passing through usable() to filter hidden entries and deduplicate across pages.

3. Progressive UI Reactivity (RepoViewModel.kt & RepoScreen.kt)

  • As each page completes, _catalog.value is emitted with updated modules, current page, and syncProgress.
  • RepoViewModel reacts to dataSource.catalog via Kotlin Coroutines combine, mapping to allEntries and applying search query filtering and sorting off the main thread (Dispatchers.Default). Newly ingested modules appear live in the list as each page arrives.
  • Client-side Filtering & Sorting: With all active modules populated directly into the repository catalogue, search filtering and sorting (recently updated, name, and star count) evaluate cleanly against the comprehensive in-memory model without requiring separate search index endpoints.

4. UI Indicators and Interaction Locking

  • Added a sync indicator (LinearProgressIndicator) and status label above the bottom navigation bar showing sync progress percentage and page count while catalog.isSyncing is active.
  • Filter and sort selection is disabled while sync is in progress (enabled = !isSyncing) to prevent users from sorting partial datasets before all pages arrive.

5. 1-Hour In-Memory Cache and Tab Navigation

  • Tracked lastFullSyncTimeMillis upon completion of full catalog sync.
  • In RepoRepository.refresh(), if force == false and the existing catalogue is fully synced within CATALOG_MAX_AGE_MINUTES (60 minutes), the network sync is skipped.
  • This eliminates unnecessary automatic resyncs and progress bar flashes when switching between bottom navigation tabs.
  • Manual pull-to-refresh (force == true) bypasses the cache, enforces CacheControl.FORCE_NETWORK, and resets the 1-hour window.

@mubashardev
mubashardev force-pushed the feat/store-paginated-sync branch from d344aee to 647a8f3 Compare October 20, 2026 09:37
@mubashardev mubashardev changed the title store: paginate catalogue sync and update mirror endpoints store: paginate catalogue sync, enable progressive UI population, and add 1-hour cache Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant