Skip to content

Build a resilient Last.fm importer and reconcile cross-device plays - #44

Draft
rianjs wants to merge 8 commits into
mainfrom
feature/43-parallel-lastfm-history-download
Draft

Build a resilient Last.fm importer and reconcile cross-device plays#44
rianjs wants to merge 8 commits into
mainfrom
feature/43-parallel-lastfm-history-download

Conversation

@rianjs

@rianjs rianjs commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Download full Last.fm history in cached, resumable four-page windows, checkpointed oldest-to-newest, and aggregate only after the complete source snapshot is present.
  • Keep Spotify matching lazy to the visible review batch; preserve keyboard navigation, independent scrolling, match-confidence guidance, opaque dialogs, and bounded queue rendering.
  • Persist accepted Retune scrobble receipts and reconcile them as a multiset so local plays are not counted twice when Last.fm returns them.
  • Apply mapped external/mobile scrobbles additively and exactly once; unresolved plays remain resumable review work while later sync windows continue.
  • Preserve account-bound track/album mappings and permanent ignore rules; explicit track mappings win and temporary skips remain temporary.
  • Establish the first incremental checkpoint at activation time, then sync on launch, reconnect, or the explicit Preferences action with no periodic polling.
  • Journal the exact before/after application boundary and quarantine unreadable or unsupported sync/mapping state before fresh persistence.
  • Export reusable mappings only; keep checkpoints, receipts, caches, journal, and backlog machine-local.

Closes #43
Closes #46
Tracks #45

Automated checks

  • cargo fmt --all --check — passed.
  • Focused Last.fm Rust suite — 125 passed.
  • cargo clippy --workspace --all-targets -- -D warnings — passed; the existing block v0.1.6 future-incompatibility warning remains.
  • cargo test --workspace — passed; 488 tests passed and one real-audio-device test was intentionally ignored.
  • node scripts/check-docs.mjs — passed.
  • Desktop npm test — 54 passed.
  • Desktop npm run lint — passed with 7 pre-existing exhaustive-deps warnings.
  • Desktop TypeScript check and production build — passed.
  • git diff --check — passed.
  • GitHub CI — all 7 checks passed.

Empirical validation

  • Real 23,139-batch / 246,009-play session rendered and remained responsive.
  • Four native resize cycles repainted in 0.84–1.39s; CPU peaked at 21.7% instead of pinning for tens of seconds.
  • Queue switching took 0.59s; Up-arrow navigation took 0.48s.
  • A real 24-hour incremental sync produced 69 unresolved plays; adding one persisted mapping applied exactly one external play, then a restart left the count unchanged.
  • The application journal cleared, the original Retune profile was restored byte-for-byte, and exact paginated Spotify snapshots confirmed all 101 saved albums and 1,473 saved tracks were unchanged.

This PR remains intentionally draft on the long-lived importer branch; it is not ready to merge.

@rianjs-bot rianjs-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated PR Review

Reviewed commit: 4bd016534abd
Profile: codex-rianjs-bot - Posting as: rianjs-bot[bot]

Summary

Reviewer Findings
architecture:seams 0
frontend:view-state 1
rust:implementation-tests 0
tauri:config-ipc 0
security:credential-boundary 0
frontend:view-state (1 finding)

Major - apps/desktop/test/ui.test.ts:231

This source-text test does not exercise the changed async UI state. In particular, it misses the A→B race where A settles after B: openQueueItem unconditionally clears pageLoading in A's finally, even though B is still loading. The queue becomes enabled and the UI can replace the matching status with “No review page selected” before B resolves. Add a deterministic deferred-promise regression test that tracks loading through overlapping requests and asserts only the newest request may clear it; then guard the production cleanup by the request generation.

Reviewer Coverage

  • architecture:seams — complete (broad); inspected 11 assigned files (12 inspected across reviewers): ARCHITECTURE.md, apps/desktop/src-tauri/src/lastfm_import.rs, apps/desktop/src/App.css, apps/desktop/src/App.tsx, apps/desktop/src/LastFmImporter.tsx, apps/desktop/src/appState.ts, apps/desktop/src/lastfmImportState.ts, apps/desktop/src/lastfmImporter.css, apps/desktop/src/types.ts, docs/architecture/library.md, docs/architecture/persistence.md; skipped: none; constraints: Review was limited to ownership, dependency direction, lifecycle boundaries, provider isolation, persistence seams, and proportionate abstraction; implementation, Tauri IPC, credential, and React behavior defects were out of scope.
  • frontend:view-state — complete (constrained); inspected 1 assigned file (12 inspected across reviewers): apps/desktop/test/ui.test.ts; skipped: none; constraints: Narrow review limited to the assigned UI test file; production UI was consulted only to assess whether the added regression coverage exercises its changed behavior.
  • rust:implementation-tests — complete (broad); inspected 1 assigned file (12 inspected across reviewers): apps/desktop/src-tauri/src/lastfm_import.rs; skipped: none; constraints: Focused Rust test execution could not be run because the cargo executable is unavailable in this review environment.
  • tauri:config-ipc — complete (constrained); inspected 2 assigned files (12 inspected across reviewers): apps/desktop/src-tauri/src/lastfm_import.rs, apps/desktop/test/ui.test.ts; skipped: none; constraints: Focused Rust test command could not run because the required RTK proxy could not spawn cargo in this environment. Narrow Tauri IPC/configuration review only; general Rust and frontend-state concerns were out of scope.
  • security:credential-boundary — complete (constrained); inspected 1 assigned file (12 inspected across reviewers): docs/architecture/persistence.md; skipped: none; constraints: Review scope was limited to the assigned changed architecture document; implementation files were not inspected.
Inspected files (12)
  • ARCHITECTURE.md
  • apps/desktop/src-tauri/src/lastfm_import.rs
  • apps/desktop/src/App.css
  • apps/desktop/src/App.tsx
  • apps/desktop/src/LastFmImporter.tsx
  • apps/desktop/src/appState.ts
  • apps/desktop/src/lastfmImportState.ts
  • apps/desktop/src/lastfmImporter.css
  • apps/desktop/src/types.ts
  • apps/desktop/test/ui.test.ts
  • docs/architecture/library.md
  • docs/architecture/persistence.md

0 PR discussion threads considered. 0 summarized; 0 resolved.


Completed in 3m 24s | gpt-5.6-sol, gpt-5.6-terra | cr 0.10.288
Field Value
Model gpt-5.6-sol, gpt-5.6-terra
Reviewers architecture:seams, frontend:view-state, rust:implementation-tests, tauri:config-ipc, security:credential-boundary
Engine codex_cli · gpt-5.6-sol, gpt-5.6-terra
Reviewed by cr · rianjs-bot[bot]
Duration 3m 24s wall · 9m 04s compute
Cost unavailable
Tokens 2.5M in / 23.3k out

Per-workstream usage

  • orchestrator-selection — gpt-5.6-terra
    • In: 18.3k
    • Out: 462
    • Cache read: 11.0k
    • Cache create: unavailable
    • Cost: unavailable
    • Duration: 11s
  • architecture:seams — gpt-5.6-sol
    • In: 912.7k
    • Out: 6.5k
    • Cache read: 807.4k
    • Cache create: unavailable
    • Cost: unavailable
    • Duration: 2m 57s
  • frontend:view-state — gpt-5.6-terra
    • In: 262.2k
    • Out: 4.6k
    • Cache read: 201.2k
    • Cache create: unavailable
    • Cost: unavailable
    • Duration: 1m 33s
  • rust:implementation-tests — gpt-5.6-terra
    • In: 521.1k
    • Out: 4.8k
    • Cache read: 460.0k
    • Cache create: unavailable
    • Cost: unavailable
    • Duration: 1m 44s
  • tauri:config-ipc — gpt-5.6-terra
    • In: 685.2k
    • Out: 5.1k
    • Cache read: 596.2k
    • Cache create: unavailable
    • Cost: unavailable
    • Duration: 1m 52s
  • security:credential-boundary — gpt-5.6-terra
    • In: 107.9k
    • Out: 1.2k
    • Cache read: 71.7k
    • Cache create: unavailable
    • Cost: unavailable
    • Duration: 38s
  • orchestrator-rollup — gpt-5.6-terra
    • In: 38.9k
    • Out: 557
    • Cache read: 28.2k
    • Cache create: unavailable
    • Cost: unavailable
    • Duration: 6s

assert.match(appCss, /\.modal-backdrop \{[^}]*z-index: 10;[^}]*background: rgb\(0 0 0 \/ \.38\)/s)
})

test('Last.fm queue selection keeps refresh stable and controls retain native button semantics', () => {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This source-text test does not exercise the changed async UI state. In particular, it misses the A→B race where A settles after B: openQueueItem unconditionally clears pageLoading in A's finally, even though B is still loading. The queue becomes enabled and the UI can replace the matching status with “No review page selected” before B resolves. Add a deterministic deferred-promise regression test that tracks loading through overlapping requests and asserts only the newest request may clear it; then guard the production cleanup by the request generation.

Reply inline to this comment.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged — fixed in commit 91417de.

Root cause: loadSelectedImportPage owned request generations, but LastFmImporter.openQueueItem unconditionally cleared pageLoading in its outer finally. A stale request could therefore clear loading while a newer B request was still pending.

The helper now owns current-generation-only completion in a finally path, including errors. openQueueItem passes setPageLoading(false) to that completion callback, while the existing current-generation error check remains unchanged.

Regression coverage:

  • The existing deterministic deferred A→B test now starts A and B, resolves A while B is pending, and asserts loading remains true.
  • Resolving B asserts loading becomes false and B remains selected/applied.
  • The test was red before the fix and green after it.

Checks:

  • npm test: 51 passed, 0 failed.
  • npx tsc --noEmit -p tsconfig.app.json: no errors.
  • npm run lint: passed; 7 pre-existing exhaustive-deps warnings.
  • npm run build: passed.
  • git diff --check: passed.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks—the generation-scoped completion path and deferred A→B regression cover the reported race.

Summary:
Fixed stale-request loading cleanup: only the current request generation may clear page loading, with deterministic overlapping-request regression coverage.

@rianjs-bot rianjs-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated PR Review

Reviewed commit: 91417de4189b
Profile: codex-rianjs-bot - Posting as: rianjs-bot[bot]

Summary

Reviewer Findings
architecture:seams 0
frontend:view-state 0
rust:implementation-tests 0
tauri:config-ipc 0
security:credential-boundary 0

Reviewer Coverage

  • architecture:seams — complete (constrained); inspected 11 assigned files (12 inspected across reviewers): ARCHITECTURE.md, apps/desktop/src-tauri/src/lastfm_import.rs, apps/desktop/src/App.css, apps/desktop/src/App.tsx, apps/desktop/src/LastFmImporter.tsx, apps/desktop/src/appState.ts, apps/desktop/src/lastfmImportState.ts, apps/desktop/src/lastfmImporter.css, apps/desktop/src/types.ts, docs/architecture/library.md, docs/architecture/persistence.md; skipped: none; constraints: Review was limited to ownership, dependency direction, lifecycle boundaries, provider isolation, persistence seams, and proportionate abstraction; implementation, Tauri IPC, credential, and React behavior defects were out of scope. Review was read-only; verification relied on source inspection and the supplied automated-check results.
  • frontend:view-state — complete (constrained); inspected 1 assigned file (12 inspected across reviewers): apps/desktop/test/ui.test.ts; skipped: none; constraints: Narrow review limited to the assigned UI test file.
  • rust:implementation-tests — complete (constrained); inspected 1 assigned file (12 inspected across reviewers): apps/desktop/src-tauri/src/lastfm_import.rs; skipped: none; constraints: none
  • tauri:config-ipc — complete (constrained); inspected 2 assigned files (12 inspected across reviewers): apps/desktop/src-tauri/src/lastfm_import.rs, apps/desktop/test/ui.test.ts; skipped: none; constraints: Narrow Tauri IPC/configuration review; the follow-up change is confined to frontend test coverage.
  • security:credential-boundary — complete (constrained); inspected 1 assigned file (12 inspected across reviewers): docs/architecture/persistence.md; skipped: none; constraints: Review scope was limited to the assigned changed architecture document; implementation files were not inspected.
Inspected files (12)
  • ARCHITECTURE.md
  • apps/desktop/src-tauri/src/lastfm_import.rs
  • apps/desktop/src/App.css
  • apps/desktop/src/App.tsx
  • apps/desktop/src/LastFmImporter.tsx
  • apps/desktop/src/appState.ts
  • apps/desktop/src/lastfmImportState.ts
  • apps/desktop/src/lastfmImporter.css
  • apps/desktop/src/types.ts
  • apps/desktop/test/ui.test.ts
  • docs/architecture/library.md
  • docs/architecture/persistence.md

1 PR discussion threads considered. 1 summarized; 1 resolved.


Completed in 2m 15s | gpt-5.6-sol, gpt-5.6-terra | cr 0.10.288
Field Value
Model gpt-5.6-sol, gpt-5.6-terra
Reviewers architecture:seams, frontend:view-state, rust:implementation-tests, tauri:config-ipc, security:credential-boundary
Engine codex_cli · gpt-5.6-sol, gpt-5.6-terra
Reviewed by cr · rianjs-bot[bot]
Duration 2m 15s wall · 3m 09s compute
Cost unavailable
Tokens 4.3M in / 29.3k out

Per-workstream usage

  • architecture:seams — gpt-5.6-sol
    • In: 1.9M
    • Out: 9.6k
    • Cache read: 1.7M
    • Cache create: unavailable
    • Cost: unavailable
    • Duration: 1m 39s
  • frontend:view-state — gpt-5.6-terra
    • In: 510.2k
    • Out: 5.9k
    • Cache read: 425.7k
    • Cache create: unavailable
    • Cost: unavailable
    • Duration: 30s
  • rust:implementation-tests — gpt-5.6-terra
    • In: 673.4k
    • Out: 5.4k
    • Cache read: 595.7k
    • Cache create: unavailable
    • Cost: unavailable
    • Duration: 16s
  • tauri:config-ipc — gpt-5.6-terra
    • In: 898.5k
    • Out: 5.8k
    • Cache read: 792.3k
    • Cache create: unavailable
    • Cost: unavailable
    • Duration: 20s
  • security:credential-boundary — gpt-5.6-terra
    • In: 199.8k
    • Out: 1.9k
    • Cache read: 148.0k
    • Cache create: unavailable
    • Cost: unavailable
    • Duration: 17s
  • orchestrator-rollup — gpt-5.6-terra
    • In: 87.4k
    • Out: 725
    • Cache read: 70.7k
    • Cache create: unavailable
    • Cost: unavailable
    • Duration: 4s

@rianjs rianjs changed the title Parallelize Last.fm history imports Build a resilient Last.fm importer and reconcile cross-device plays Aug 21, 2026
@rianjs

rianjs commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator Author

Test coverage assessment — PR #44

Major

  • apps/desktop/src-tauri/src/lastfm_import.rs:3977-4134 — The exactly-once application/recovery protocol has no test that drives either apply_completed_incremental_range or recover_pending_incremental_journal (the only journal test, incremental_updates_saturate_and_journal_recovery_is_exactly_once at :6375-6414, calls the pure helper). Add a crash-point test with persisted sync state and receipt ledger for both library-is-before and library-is-after recovery; otherwise wrong checkpoint/backlog finalization or premature receipt pruning can pass every current test while causing duplicate counts or loss in production.

  • apps/desktop/src-tauri/src/lastfm_import.rs:4137-4348 — There is no test invocation of the incremental runner/command. incremental_cache_resumes_and_filters_the_padded_query_window (:6673-6762) manually seeds checkpoints and reads the cache, so it does not prove first activation sets syncedThrough without requesting history, a later manual/launch/reconnect sync uses the padded fixed range, or source download makes no Spotify calls. A small end-to-end fake Last.fm/Spotify test should cover those boundaries; the current tests would pass if this command stopped downloading, backfilled on first use, or touched Spotify during download.

  • apps/desktop/src-tauri/src/lastfm.rs:1342-1407 — Receipt parsing is tested at :2323-2343 and pruning at :2288-2320, but flush_once itself is not exercised. Add a fake accepted/rejected batch test that reloads lastfm-scrobbles.json and verifies only accepted code-0 results survive as durable corrected/submitted receipts. As written, a regression that drops new_receipts, persists them with the wrong queue revision, or records receipts for rejected results can still pass.

Minor

  • apps/desktop/src-tauri/src/lastfm_import.rs:1874-1900,1941-1960 — Backup coverage stops at envelope serialization (apps/desktop/src-tauri/src/lib.rs:4128-4157). It never restores mappings through Service, verifies they remain dormant for mismatched identities, or verifies they activate and are reused only after an exact Last.fm/Spotify identity match. Add that lifecycle test; a faulty account guard would currently pass the backup round-trip test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reconcile incremental Last.fm plays across devices Download Last.fm history pages concurrently

1 participant