Custom-extraction ingest: advanced-form overrides + raw bookmarklet path - #63
Merged
Merged
Conversation
- Remove PWA requirement for WiFi sync feature since Network Information API works in Chrome mobile browser, not just PWAs - Fix syncEnabled default to true (was inconsistently false vs RemoteStorageProvider which defaults to true) - Add network.ts with utilities for detecting network type and PWA mode - Add SyncStatusProvider to manage sync status based on WiFi connection - Add WiFi-only sync toggle to preferences (shown when network info API is supported) - Add sync status indicator to article list (green cloud when active, orange cloud-off when paused on cellular) - Add Network Connection Information and Current Status sections to diagnostics page for debugging
Fix WiFi sync indicator for mobile browsers
Problem: - When the app opens and connects to remote storage, documents become unavailable during the resync process - This was caused by the IndexedDB being cleared on 'disconnected' events, which can fire during normal reconnect cycles when the app starts Root Cause: - The 'disconnected' event handler unconditionally cleared the entire IndexedDB (db.articles.clear()) - During app startup, remoteStorage.js may trigger disconnect/reconnect cycles, causing the database to be cleared before sync completes - Users trying to access documents during this window would find them unavailable Solution: - Added sync state tracking (isSyncing, hasCompletedInitialSync flags) - Modified disconnected event handler to only clear database on user-initiated disconnects - Preserves local database during sync/reconnect cycles - Added enhanced logging with emoji indicators to track connection lifecycle - Logs article count and sync state when disconnect occurs for debugging This ensures documents remain available in IndexedDB during automatic resync operations while still allowing proper cleanup on user disconnect. Related: remoteStorage.js Issue #939 (race conditions during disconnect)
Improvements made to all storage event handlers:
1. Error Handler
- Removed all commented-out dead code and unused variables
(lastNotificationTime, lastSyncErrTime, notificationTimeout)
- Added sync state reset on error to prevent stuck states
- Added emoji logging (🚨) for consistency
2. Network Events
- Changed from console.debug to console.info for visibility
- Added emoji indicators (📴 offline, 📶 online)
- Added sync state management in network-online handler
- Properly set isSyncing flag when network reconnects
3. Sync Request Done Handler
- Added emoji logging (🔄)
- Now properly resets isSyncing flag for ongoing syncs
- Improved comments about incremental processing
4. Wire Events (NEW)
- Added wire-busy handler to track network activity start
- Added wire-done handler to track network activity end
- Uses debug level logging with ⚡ emoji
5. Change Event Handler
- Added structured logging with only article-relevant changes
- Added emoji indicators (🔄, 📥, 🗑️, ⏭️, ✅, ❌)
- Added try-catch error handling around processArticleFile
- Retry logic: removes failed articles from processedSet
- Better indentation for nested log messages
- More informative log messages
All changes improve observability, error handling, and sync state
management without altering core functionality.
Problem: - When users pull down on the page (common mobile gesture), the browser's native pull-to-refresh triggers a full page reload - Page reload causes remoteStorage to reconnect and resync all data - This makes documents temporarily unavailable during the resync Solution: - Added `overscroll-behavior-y: contain` to html and body elements - This CSS property disables the browser's pull-to-refresh gesture - Prevents accidental page reloads while maintaining scrolling behavior - Works on Chrome, Firefox, Opera, and modern mobile browsers Browser Compatibility: - Chrome/Edge: Full support - Firefox: Full support - Safari: Supported in iOS 16+ - Opera: Full support The 'contain' value prevents the browser from triggering navigation actions (like pull-to-refresh) when the user scrolls past the boundary of the scrolling area.
Problem: - Articles took 10+ seconds to load when online but loaded instantly in airplane mode - The issue was remoteStorage.js getFile() making network requests or waiting for network timeouts when online before returning content - Default maxAge behavior is 2*syncInterval when connected, causing network checks even for cached content Root Cause: - ArticleScreen.tsx was calling storage.client.getFile() without maxAge parameter - When online, getFile() attempted network validation/refresh - When offline, it immediately returned from local cache (fast) - This affected article HTML, thumbnails, and storage size calculations Solution: - Added maxAge: false parameter to all getFile() calls for reading content (not syncing) - maxAge: false forces local-only reads without network requests - Per remoteStorage.js docs: "If the maxAge requirement is set to false, the promise will always be fulfilled with data from the local store" Changes: 1. ArticleScreen.tsx (lines 313, 323, 182): - Article HTML loading (cleaned and original views) - HTML loading for metadata editing 2. storage.ts (lines 429, 476): - Storage size calculations (total and per-article) 3. tools.ts (line 143): - Thumbnail loading Impact: - Articles now load instantly when online (same speed as offline) - No more 10-second "loading content..." delays - All reads use local cache; sync still updates cache in background - Network bandwidth saved by avoiding unnecessary validation requests Note: The processArticleFile() getFile call (storage.ts:118) was NOT changed because it's used during sync operations and needs to fetch from remote storage.
…s-IVKwK Fix documents becoming unavailable during remote storage resync
The "Save raw content" form derived all metadata from Readability, so a user pasting clean content had no way to set the title, author, or source URL. Add optional fields for these that override Readability's output when provided and fall back to it when blank. - ingestion: add IngestOverrides threaded through ingestHtml -> readabilityToArticle; a provided title also recomputes the slug so the stored folder name matches - SubmitScreen: add optional Title/Author/URL fields; the URL is also passed through so relative image paths resolve - tests: cover override, trimming, blank fallback, and no-op cases Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
✅ Deploy Preview for savrdev ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Adds a Readability-bypassing ingest path for content Savr can't extract itself. A site-specific bookmarklet does the extraction and hands Savr the finished body + metadata, which are stored verbatim. The canonical case is a YouTube transcript: it's injected by the page's JS (absent from the statically-fetched HTML) and Readability would reject it. - ingestion: extract the shared downstream (image download, render, summary, store/sync) into persistArticle; ingestHtml now calls it, and a new ingestRaw builds the Article directly from supplied content/metadata with no Readability. Raw body is run through convertToHtml so plain/markdown/ html all work. - ArticleListScreen: new savr-raw message flow behind ?rawIngest=1, with a savr-ready handshake so the bookmarklet posts its payload exactly once; queues if the RS client isn't ready yet, mirroring the savr-html path. - bookmarklet: savr-youtube-transcript.unminified.js — opens the transcript panel, scrapes segments + title/channel, posts a savr-raw payload. - tests: ingestion-raw.test.ts proves the bypass (Readability mock never wins), metadata trimming, file writes for sync, null-client safety, and content-type auto-detect. - README: document the raw-ingest contract and the example bookmarklet. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jonocodes
changed the title
Renders a second draggable bookmarklet next to the regular one, generated inline with the instance origin injected (mirroring the existing bookmarklet link). Clicking it on a YouTube watch page scrapes the transcript and sends it to Savr via the savr-raw path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Records two items already merged to dev but missing from the changelog: the non-intrusive article-list search (#65) and the fix for the bookmarklet URL param re-triggering ingestion on reload. PR #63's own work is added after it merges, per the file's convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
9 tasks
YouTube replaced <ytd-transcript-segment-renderer>/.segment-* with
<transcript-segment-view-model> and ytw* class names, so the bookmarklet
found no segments and always alerted "Could not open the transcript panel"
even when it was open. Match both old and new selectors, and drop the
seg.textContent fallback — the new DOM nests the timestamp and a hidden
a11y label ("1 second") inside each segment, which the fallback would
have scooped into the transcript text.
Verified against a live YouTube page with playwright-core: the extractor
opens the panel and returns clean lines + title.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The e2e-smoke job's "Install Playwright browsers" step ran `playwright install --with-deps`, whose `apt-get update` hit the dl.google.com chrome-stable source preinstalled on ubuntu-latest. That repo was serving a corrupt Packages.gz (hash-sum mismatch), failing the whole update even though we only need Playwright's bundled Chromium from Ubuntu repos. Remove that source list first so apt update succeeds. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
playwright.config's webServer wrapped `npm run dev` in `flox activate`, which works locally but exits 127 on CI (no flox on the GitHub runner), so the e2e-smoke webServer failed to start. Use the plain command when CI is set (matching dev's config), keep the flox wrapper locally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The bookmarklet lived twice: in bookmarklet/savr-youtube-transcript.unminified.js and as an inline minified copy in PreferenceScreen.tsx. The copies had already drifted — the inline one had lost the new timestamp-selector handling. To keep one hand-edited source: - the unminified file now uses a "__SAVR_ORIGIN__" placeholder (substituted with the app's real origin at runtime); - a small Vite plugin (vite.config.ts) minifies the file with esbuild at build/dev time and exposes it via <file>?bookmarklet; - PreferenceScreen.tsx imports that and wires it onto the preferences link, replacing ~20 lines of duplicated inline script. Verified by simulating the exact plugin+substitution pipeline and running the resulting bookmarklet end to end against a fixture page reproducing YouTube's new transcript-segment-view-model DOM: panel auto-open, clean extraction (no timestamps, no a11y pollution), correct title/author/url, and a successful savr-ready -> savr-raw handshake.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds the primitives for client-side custom extraction — a bookmarklet does site-specific extraction and hands Savr finished content + metadata, so Savr never has to figure out extraction itself. Canonical use case: a YouTube transcript, which is injected by the page's JS (absent from the statically-fetched HTML) and which Readability would reject.
Two layers, groundwork first:
1. Metadata overrides on the advanced "Save raw content" form
The form derived all metadata from Readability. Added optional Title / Author / URL fields that override Readability's output when provided and fall back when blank (fully backward-compatible). A provided title recomputes the slug so the stored folder matches.
ingestHtmlgained an optional trailingIngestOverridesparam — existing callers unaffected.2. Raw-ingest path (
ingestRaw+savr-raw) — bypasses Readabilitylib/src/ingestion.ts— extracted the shared downstream (image download/resize, reader render, optional AI summary, store + sync) intopersistArticle.ingestHtmlnow calls it; newingestRawbuilds theArticledirectly from supplied{content, contentType, title, author, url}with no Readability — mirroring howingestPdf/ingestImagealready construct Articles. The body still runs throughconvertToHtml, so plain/markdown/html all work.src/components/ArticleListScreen.tsx— newsavr-rawmessage flow behind?rawIngest=1. Because a bookmarklet can't know when the app has mounted, the app pings the opener withsavr-readyon an interval until the payload arrives; the bookmarklet replies once. Queues the payload if the RemoteStorage client isn't ready yet, mirroring the existingsavr-htmlpath. The Save/URL controls stay disabled during raw ingest so nothing misfires.bookmarklet/savr-youtube-transcript.unminified.js— adapts a working transcript scraper: opens the transcript panel, scrapes segments + video title + channel, posts asavr-rawpayload.SAVR_ORIGINand aSTRIP_TIMESTAMPStoggle at the top.Tests
ingestion-raw.test.ts— proves the bypass (a distinctive Readability mock never wins), metadata trimming/null-defaults, all sync files written, null-client safety, content-type auto-detect.tsc --noEmitclean.Notes
ingestRawis a separate, opt-in path.savr-rawcontract is generic — any future site-specific bookmarklet reuses it; YouTube is just the first consumer.🤖 Generated with Claude Code