feat(settings): data recovery for clobbered lists by dmnyc · Pull Request #22 · DocNR/jank

dmnyc · 2026-09-25T04:04:09Z

Summary

Adds Settings → Data recovery: scan relays for older versions of your lists and profile (follow list, mute list, profile, bookmarks, NIP-4e key list, relay lists, blocked relays) and restore one if a client clobbered it. Scans are read-only, and nothing is published until you review the change and confirm. This implements the Lazarus spec (0.5.0-draft): recovery is never automatic, everything found is shown, and publishing goes through your own signer, one event per click.

Where it looks

Most relays keep only the latest version of a replaceable event, so the scan covers every relay in your relay list (read and write), the default relays, and an archival set led by relay.ditto.pub, which keeps every version, and hist.nostr.land. Relays that return a full page of 50 versions can be paged further back with Load older versions. On my account, my first five outbox relays held 1 version of my follow list; the first scan finds 51, and paging back finds all 439.

What gets recommended

Lists shrink through normal curation, so a bigger older version isn't a reason to restore. A version is recommended only when the current one looks clobbered: a sudden drop of at least a fifth of the list (and at least 5 items) between two versions, which the list hasn't recovered from. Drops within a day of each other count as one episode, and the recommendation is the fullest version from before it. A clobber the list has since been edited on at least 5 times over a week is settled, so nothing is recommended: my follow list went through a clobber fight in mid-June, but I've edited it 161 times since, and the page correctly recommends nothing. When there is a recommendation, it's pinned above the list.

Long histories

Newest first, the list folds into groups: runs of small edits, and clobber episodes marked "sudden drops". Only the current version, and empty versions when shown, keep their own rows: my follow list's first scan shows 2 rows instead of 51, and all 439 versions fit in 8. Past empty versions are hidden until you ask for them and are never offered for restore, so a clobbered state can't be put back by accident. Sorting by size lists every version.

Private items

Mute lists and bookmarks can keep items encrypted in content (NIP-51). Counts include them: exact once decrypted (reusing content the app already decrypted), otherwise estimated from the encrypted size (NIP-44 padding or NIP-04 block size). That estimate is enough to tell an emptied private list from a full one, which public tags alone read as the same zero.
Only rows shown on their own and the pinned recommendation are decrypted up front; grouped versions are decrypted when their group is expanded or a version is reviewed, one at a time.
The restore review includes private items, so the list of accounts that would be unmuted is accurate.
Legacy relay JSON in kind 3 content isn't mistaken for private items.

Remote signers

NIP-46 requests are NIP-44 encrypted, which caps them at 65,535 bytes, about 850 follows. Since Jank accounts are usually paired over NIP-46, the page says this up front for bunker accounts. Versions too large to restore are marked in the list, and restoring one is blocked with an explanation that large lists need a NIP-07 extension or a local key. Grouped versions are decrypted only when you review one, so long histories don't flood the signer with requests. The size checks live in src/lib/nip46.ts so other publishing paths can use them.

Reviewing a restore

The review lists the items that would be added and removed, comparing them by type and value, so a rewritten relay hint or petname isn't a change (relay lists keep the read/write marker). Profile restores list the fields that change. View-only accounts can scan but aren't offered a restore.

Publishing

Right before signing, the current version is re-read from jank's own copy and your write relays. If it changed since the review opened, the review is compared again and has to be confirmed again.
The restored event is dated after the version it replaces, max(now, current.created_at + 1), and only the list's author can restore it.
Success is judged on your write relays. Every other relay that answered the scan gets the restored version as a best effort, since those relays hold older copies of the list and would keep serving the clobbered one.
Afterward, jank's own copy of the list is updated, so its next follow or mute builds on the restored version.

Notes

Opens in a detail column at /settings/data-recovery. The Settings entry shows for signed-in accounts and acts as the active account through useNostr(), like the other settings pages.
Your own profile shows a Restore link in its stats row (where other profiles show who follows them) that opens Data recovery with the follow list selected.
While open, the page reserves its column's scrollbar space, so opening or folding a group doesn't shift the controls.
Relay reads go through client.subscribe (closed as soon as a relay finishes or times out) and relayListService. The archival set is kept to the six relays observed holding history.
New strings are appended to en.ts only; other locales fall back to English.
No release-notes entry or version bump. Happy to add one if you want it in What's new.

Testing

70 data recovery tests: real NIP-44 and NIP-04 payloads of 0 to 593 items, the scan and publish relay sets, paging (including relays that ignore until), clobber episodes and settling, grouping, empty versions, sorting, NIP-46 request sizes, the restore timestamp, item comparison, profile field changes, the pre-sign re-check and relay timeouts. The full suite (973 tests), npm run build, npm run lint and tsc -b pass.
Scanned my own account in a dev build: 439 follow-list versions fold into 8 rows with three clobber episodes marked, and nothing is recommended since I've edited the list since each of them.
Signed in with a remote signer: versions too large to restore are marked in the list.

Scan relays for historical versions of replaceable data kinds (follows, mutes, profile, bookmarks, relay lists, NIP-4e key list), rank candidates per kind, and restore a chosen version on an explicit user click. Per-kind semantics live in one registry table; the scan/rank/delta core never hardcodes kinds. A user's own relays usually keep only the latest version, so the scan covers every relay in the user's relay list (read and write), the default relays, and an archival set led by hist.nostr.land and relay.ditto.pub, which keep full history. Relays that fill a page of 50 can be paged further back with "Load older versions". A version is recommended only when the current one looks clobbered: a sudden drop of at least a fifth of the list, and at least 5 items, that it hasn't recovered from. Drops within a day of each other count as one episode, and the recommendation is the fullest version from before it. Slow curation never triggers one, and neither does a clobber the list has since been edited on at least 5 times over a week. Empty versions are never recommended, and for kinds where empty is a meaningful state nothing is ranked at all and the user chooses with intent. The recommended version is pinned above the list, and list kinds can be sorted by date (the default) or size. Newest first, the list folds into groups: runs of small edits, and clobber episodes marked "sudden drops", so a history of hundreds of versions shows a handful of rows. Only the current version, and empty versions when shown, keep their own rows. Past empty versions are hidden until asked for and never offered for restore, so a clobbered state can't be put back by accident. Only versions on their own rows and the pinned recommendation are decrypted up front, grouped ones when their group is expanded or a version is reviewed, and decryptions run one at a time. Private list items (NIP-51) count toward every version: exactly once decrypted, reusing content the app already decrypted, and otherwise estimated from the encrypted payload size, which is enough to tell an emptied private mute list from a full one. The restore review includes private items, so unmutes are listed accurately. NIP-46 requests are NIP-44 encrypted and capped at 65,535 bytes, so versions too large for a remote signer are only sized, and restoring them points to a NIP-07 extension or a local key. On a remote signer, versions too large to restore are marked in the list, and grouped versions are decrypted only when reviewed, so long histories don't flood the signer with requests. Restores publish to all of the user's write relays plus every relay that answered the scan, so the restored version replaces the old one wherever the scan found it. The screen lives at /settings/data-recovery and is linked from Settings for signed-in accounts. While open it reserves its column's scrollbar space, so opening or folding a group doesn't shift the controls sideways. New strings are English only. Implements the Lazarus draft spec 0.4.0 (https://github.com/dmnyc/lazarus): recovery is never automatic, everything found is shown, and publishing always goes through the user's own signer, one event per click.

DocNR

Thanks for this, it's a really useful feature and the scan, ranking and grouping logic is careful work. I went through it and have some points before merging. Details are in the inline comments.

Blocking (a restore can be undone or lost):

After a restore, jank's local list cache still holds the clobbered version, so jank's own next follow or mute rebuilds from it and clobbers the list again.
The restored event is stamped "now", which can be older than a clobbered version with a skewed clock. The restore then loses on relays and in the cache.
The restore compares against the list as it was at scan time. Edits made after the scan (for example a follow from another column) are dropped without a warning.

Should fix:
4. No check that the list being restored belongs to the account that signs it.
5. The wide publish set (every relay that answered the scan) can make a successful restore show as "Could not publish", because publishEvent needs a third of the relays to accept.
6. A profile (kind 0) restore shows "0 added, 0 removed", so the user can't see what they're restoring.
7. The delta treats a p tag that only changed its relay hint or petname as both added and removed.

Smaller:
8. preparingReview isn't cleared in a finally.
9. View-only npub accounts are offered Restore.
10. Timed-out relay queries aren't cancelled.

Review drafted with Claude Code.

DocNR · 2026-09-25T12:36:51Z

+    try {
+      const draft = buildLazarusRecoveryDraft(pending.candidate.event)
+      const signed = await signEvent(draft)
+      await client.publishEvent(


1. Restore doesn't update jank's list cache. This publishes with client.publishEvent but never calls replaceableEventCache.updateCache(signed). jank's in-memory store and IndexedDB keep the clobbered version.

Scenario: the user restores 400 follows over a clobbered 3-follow list, then clicks Follow in any column. ScopedUserListsProvider.follow calls followListService.fetchFollowListEvent, which returns the stale IndexedDB event first. It builds 3+1 follows and publishes them, so jank clobbers the restore itself. Every other list-publish path calls updateCache(real) after publishing for this reason. The same applies to mute lists and bookmarks.

Fixed in 486b321. After publishing, the restore goes through replaceableEventCache.updateCache (or updateProfileEvent / updateRelayListEvent for kinds 0 and 10002), and the chosen version's decrypted private items are copied to the new event, so the next follow or mute builds on the restored list.

DocNR · 2026-09-25T12:36:51Z

+    kind: chosen.kind,
+    content: chosen.content,
+    tags: chosen.tags.map((tag) => [...tag]),
+    created_at: nowSeconds


2. created_at isn't guaranteed to be newer than the current version. Clobbering clients often have skewed clocks. If the clobbered event is dated a few minutes in the future, a restore stamped "now" is older. Relays keep the clobbered one, and the newer-wins gate in replaceableEventCache / IndexedDB rejects the restore too. The user sees "Recovery published" but nothing changes.

ScopedUserListsProvider has nextCreatedAt(base) = max(now, base.created_at + 1) for this reason. Pass the current version in and use the same rule. The helper is module-private today, so it may need to move to src/lib.

Fixed in 486b321. The draft is dated max(now, current.created_at + 1), where current is the newest version found in the re-check just before signing. I kept the rule in the lazarus core instead of moving nextCreatedAt, so ScopedUserListsProvider stays untouched.

DocNR · 2026-09-25T12:36:51Z

+    })
+  }, [])
+
+  const confirmRecovery = useCallback(async () => {


3. Edits made after the scan are silently dropped. The delta and the restore are based on scan.current from scan time. jank is multi-column, so a user can scan, follow a few people from a Home column, then come back and restore. The dialog doesn't list those follows as removed, and the restore overwrites them.

Suggestion: before signing, re-read the current version (for example from replaceableEventCache or a quick fetch). If it's newer than scan.current, recompute the delta and show it again.

Fixed in 486b321. Right before signing, the current version is re-read from IndexedDB and the write relays. If it changed since the review opened, the review is recomputed against it, a note says the list changed, and it has to be confirmed again.

DocNR · 2026-09-25T12:36:51Z

+    setRecovering(true)
+    try {
+      const draft = buildLazarusRecoveryDraft(pending.candidate.event)
+      const signed = await signEvent(draft)


4. No signer/pubkey match check. signEvent signs with the active account's signer. jank lets users cycle the active account (a), and scan / pending aren't reset when pubkey changes. If the active account changes while the dialog is open or a bunker approval is pending, this publishes account A's list as account B's.

A cheap guard: refuse to sign unless pending.candidate.event.pubkey === pubkey, and reset the scan when pubkey changes.

Fixed in 486b321. Restore refuses unless the version's author is the active account, the scan and any open review reset when the active account changes, and a signed event whose pubkey doesn't match isn't published.

DocNR · 2026-09-25T12:36:51Z

+  } catch {
+    writeRelays = getDefaultRelayUrls()
+  }
+  const urls = [...writeRelays, ...respondingRelays]


5. The wide publish set can report success as failure. client.publishEvent resolves only when a third of the target relays accept, otherwise it rejects. The responding set can include paid or restricted relays (nostr.wine, nostr.land, eden.nostr.land) and archives that won't accept writes from non-members. When fewer than a third accept, the page shows "Could not publish the recovery event" and skips the rescan, even though the event landed on several relays. The user will likely retry.

It also sends the user's (encrypted) mute list or DM-relay list to relays they never chose. Consider publishing to the write relays with the normal threshold, then sending to the extra responding relays as a best effort that doesn't affect the success result.

Fixed in 486b321. Success is judged on the write relays with the normal threshold, and the other relays that answered get the restored version as a best effort that doesn't affect the result. Those relays only answered because they already hold older copies of the same list, so this replaces what they'd otherwise keep serving rather than sending anything new.

DocNR · 2026-09-25T12:36:51Z

+  return JSON.stringify(tag)
+}
+
+export function computeLazarusDelta(


6. Profile (kind 0) restores have an empty review. The delta compares tags only, but kind 0 data lives in content. The dialog shows "0 items would be added · 0 items would be removed" and never shows the name, about, picture or lud16 being restored. A small field-level diff of the parsed content would fix it.

Fixed in 486b321. Profile restores list the fields that would change (name, display_name, about, picture, banner, nip05, lud16, lud06, website), with the old value struck through under the new one.

DocNR · 2026-09-25T12:36:51Z

+  privateUnknown: boolean
+}
+
+function tagIdentity(tag: string[]): string {


7. Tag identity is too strict. JSON.stringify(tag) makes ['p', pk] and ['p', pk, 'wss://…'] different items. If a client rewrote relay hints or petnames, the review shows "300 added · 300 removed" for an identical set of follows. That also skews grows / shrinks, and for mute lists it would show the "would be unmuted" warning when nobody is unmuted. Comparing on tag[0] + tag[1] would match how the lists are actually used.

Fixed in 486b321. Items now compare on type and value. The one exception is relay lists, where the read/write marker stays part of the item, since it changes what the relay is for.

DocNR · 2026-09-25T12:36:51Z

+      if (profile.privateItemTypes && needed.length > 0) {
+        setPreparingReview(candidate.event.id)
+        await decryptVersions(profile, needed, scanId)
+        setPreparingReview(undefined)


8. If decryptVersions rejects (for example an IndexedDB error outside the per-candidate try, or an error in the merge), this line never runs. preparingReview stays set, and pickCandidate returns early for every later click until the page remounts. Moving the reset into a finally fixes it.

Fixed in 486b321: the reset is in a finally now.

DocNR · 2026-09-25T12:36:51Z

          <ChevronRight className="rtl:-scale-x-100" />
        </SettingItem>
      )}
+      {!!pubkey && (


9. This shows for view-only npub accounts too. The page offers Review → Restore, and then signEvent throws, so the user sees "Could not publish" without knowing the account has no signing key. Consider hiding Restore, or explaining why it's unavailable, when account.signerType === 'npub'.

Fixed in 486b321. View-only accounts can still scan, but Restore is hidden and a note explains why.

DocNR · 2026-09-25T12:36:51Z

+      urls.map((url) => {
+        const filter = { kinds: [kind], authors: [pubkey], limit: SCAN_LIMIT }
+        const page = cursors ? { ...filter, until: cursors[url] } : filter
+        return withTimeout(eventCache.fetchEvents([url], page), SCAN_TIMEOUT_MS)


10. Minor: withTimeout rejects after 6 s but doesn't cancel the underlying query, so slow or AUTH-gated relays keep their subscriptions open after the page moves on. Each scan also opens sockets to about 18 archival relays on top of the user's and default relays, and "Load older versions" repeats it. That's not a problem alone, but we watch total socket count in jank (browser WebSocket limits with many columns).

Fixed in 486b321. The relay query now closes its subscription when it times out, and I trimmed the archival set from 18 relays to the 6 that actually showed history. Load older versions only re-queries relays that filled a page, which in practice is just relay.ditto.pub.

Addresses review on DocNR#22: - A restore updates jank's own copy of the list (the replaceable event cache, or the profile and relay-list updaters), so the next follow or mute builds on the restored version instead of clobbering it again. The chosen version's decrypted private items carry over to it. - The restored event is dated after the version it replaces, max(now, current + 1), so a clobbered version with a clock running ahead doesn't win on relays or in the cache. - Right before signing, the current version is re-read from the local copy and the write relays. If it changed since the review opened, the review is compared again and has to be confirmed again. - Only the list's author can restore it: the scan resets when the active account changes, and a signed event from another account isn't published. - Success is judged on the write relays; the other relays that answered the scan get the restored version as a best effort. - A profile restore lists the fields that change. - Items compare by type and value, so a rewritten relay hint or petname isn't a change; relay lists keep the read/write marker. - The review spinner clears in a finally. - View-only accounts can scan but aren't offered a restore. - Relay queries close their subscription when they time out, and the archival set is trimmed to the six relays observed holding history. Matches the Lazarus draft spec 0.5.0.

Implements the Lazarus spec (0.5.0-draft, github.com/dmnyc/lazarus) in place of the NIP-78 account-backup feature — the same design shipped in Sidecar's Jumble fork and proposed in DocNR/jank#22. Vendored core (src/lib/lazarus/, ported from the reference implementation in dmnyc/jank feat/lazarus-data-recovery; nothing is on npm): the 8-kind registry, NIP-51 private-item sizing from ciphertext length, the pure scan/rank/delta/draft algorithms (clobber episodes, settling, tombstone rules, tag-identity deltas), a zap SimplePool adapter with per-relay timeouts + archival relays + until-cursor paging, and the one write path — re-reads current before signing, guards the signing account, judges success on write relays, refreshes the app's local copy. 66 vitest cases port the spec's conformance vectors, plus a deep-paging bench. Settings gains a Data Recovery accordion (kind picker, scan on click only, every version with counts/estimates/found-on relays, recommended highlight, delta review with re-mute warning, shrink confirmation, 10044 intent question, NIP-46 size caps, view-only scan) with a windowed, memoized version list so deep archival histories don't freeze the page. The profile page's 'Restore' entry opens the same screen; ProfileEditModal's NIP-78 backup features are removed. Deleted: nostrBackup.ts, profileBackup.ts, NostrBackupSection.svelte, and the pre-spec Mutable recovery (followRecovery.ts + modal). Wallet NWC/Spark NIP-78 backups and all other 30078 app-data codecs are untouched.

Implements the Lazarus spec (0.5.0-draft, github.com/dmnyc/lazarus) in place of the NIP-78 account-backup feature — the same design shipped in the maintained Jumble fork (dmnyc/jumble-spark, the spec's reference implementation) and proposed in DocNR/jank#22. Vendored core (src/lib/lazarus/, ported from the reference implementation in dmnyc/jumble-spark feat/lazarus-data-recovery; nothing is on npm): the 8-kind registry, NIP-51 private-item sizing from ciphertext length, the pure scan/rank/delta/draft algorithms (clobber episodes, settling, tombstone rules, tag-identity deltas), a zap SimplePool adapter with per-relay timeouts + archival relays + until-cursor paging, and the one write path — re-reads current before signing, guards the signing account, judges success on write relays, refreshes the app's local copy. 66 vitest cases port the spec's conformance vectors, plus a deep-paging bench. Settings gains a Data Recovery accordion (kind picker, scan on click only, every version with counts/estimates/found-on relays, recommended highlight, delta review with re-mute warning, shrink confirmation, 10044 intent question, NIP-46 size caps, view-only scan) with a windowed, memoized version list so deep archival histories don't freeze the page. The profile page's 'Restore' entry opens the same screen; ProfileEditModal's NIP-78 backup features are removed. Deleted: nostrBackup.ts, profileBackup.ts, NostrBackupSection.svelte, and the pre-spec Mutable recovery (followRecovery.ts + modal). Wallet NWC/Spark NIP-78 backups and all other 30078 app-data codecs are untouched.

Your own profile shows a Restore link in its stats row, where other profiles show who follows them. It opens Data recovery with the follow list selected, and its tooltip explains what it recovers. The row can wrap in narrow columns, so the link never squeezes the counts.

Implements the Lazarus spec (0.6.0-draft, github.com/dmnyc/lazarus) in place of the NIP-78 account-backup feature — the same design shipped in the maintained Jumble fork (dmnyc/jumble-spark, the spec's reference implementation) and proposed in DocNR/jank#22. Vendored core (src/lib/lazarus/, ported from the reference implementation in dmnyc/jumble-spark feat/lazarus-data-recovery-v2; nothing is on npm): the 8-kind registry, NIP-51 private-item sizing from ciphertext length, the pure scan/rank/delta/draft algorithms (clobber episodes, settling, tombstone rules, tag-identity deltas), a zap SimplePool adapter with per-relay timeouts + archival relays + until-cursor paging, and the one write path. 89 vitest cases port the spec's conformance vectors, plus a deep-paging bench. The 0.6.0 relay-failure semantics are in full: every relay request ends answered/failed/timed-out and only an answered relay counts as having nothing; a scan no relay answered is a failed scan (versions that arrived first still shown); current is confirmed only when a write relay answered, withholding the recommendation until then; the kind-10002 lookup distinguishes found/missing/unknown and never substitutes defaults for an unknown list; profile deltas cover every field and tag, with decrypted private items in the delta; the pre-sign re-read fails closed, with a retry and an explicit, never-preselected restore-anyway override; only a newer version counts as a change; and the scan pool releases its connections. Settings gains a Data Recovery accordion (kind picker, scan on click only, every version with counts/estimates/found-on relays and per-relay outcomes, recommended highlight, delta review with re-mute warning, shrink confirmation, 10044 intent question, NIP-46 size caps, view-only scan, retry of unreachable relays) with a windowed, memoized version list and an inline delta panel under the clicked row. The profile page's 'Restore' entry opens the same screen; ProfileEditModal's NIP-78 backup features are removed. Deleted: nostrBackup.ts, profileBackup.ts, NostrBackupSection.svelte, and the pre-spec Mutable recovery (followRecovery.ts + modal). Wallet NWC/Spark NIP-78 backups and all other 30078 app-data codecs are untouched.

DocNR reviewed Sep 25, 2026

View reviewed changes

dmnyc mentioned this pull request Sep 25, 2026

Lazarus data recovery replaces NIP-78 account backup zapcooking/frontend#753

Merged

DocNR merged commit b65131a into DocNR:main Sep 26, 2026
1 check passed

DocNR mentioned this pull request Sep 26, 2026

fix(profile): only link to data recovery from the active account's profile #23

Merged

dmnyc deleted the feat/lazarus-data-recovery branch September 26, 2026 14:14

dmnyc mentioned this pull request Sep 26, 2026

fix(recovery): tell unreachable relays from empty ones, validate relay answers #24

Open

Conversation

dmnyc commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Where it looks

What gets recommended

Long histories

Private items

Remote signers

Reviewing a restore

Publishing

Notes

Testing

Uh oh!

DocNR left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dmnyc commented Sep 25, 2026 •

edited

Loading