Skip to content

fix(export): stream large bucket exports without JSON.parse - #993

Merged
ErikBjare merged 3 commits into
ActivityWatch:masterfrom
TimeToBuildBob:fix/export-large-buckets-timeout
Sep 23, 2026
Merged

ErikBjare merged 3 commits into
ActivityWatch:masterfrom
TimeToBuildBob:fix/export-large-buckets-timeout

Conversation

@TimeToBuildBob

@TimeToBuildBob TimeToBuildBob commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Large all-bucket exports (500k+ events) silently no-op on Android and desktop.

Two stacked root causes:

  1. Server buffers before responding — aw-server-rust serializes all events to a temp file before sending the first HTTP byte (even after perf(export): serialize HTTP exports with bounded event buffering aw-server-rust#677). For 500k events on a phone, serialization can take 30+ seconds, which silently exceeds axios's default timeout. No error is shown to the user.

  2. JS double-serialization — JSON.stringify(response.data, null, 2) re-parses and re-stringifies an already-valid JSON response in the browser heap, doubling peak memory use for large exports.

Reported and confirmed on 0.14.2b4: ActivityWatch/aw-android#228

Fix

Keep the authenticated axios client, skip JSON parse/pretty-print:

  • responseType: 'blob' + timeout: 300_000 (5-minute cap, not the 30s default, no JS object graph)
  • Spinner + disable the all-buckets button while any export is in flight (export_inflight counter)
  • User-visible error alert on failure instead of a silent drop
  • Android WebView: hand the URL to Android.exportFromUrl when the native bridge is present

Follow-up 3f26892 keeps Authorization: Bearer (Greptile P1) and counts in-flight exports (P2). 2005bb6 restores a finite 5-minute timeout so a stalled export cannot leave the spinner stuck. The original fetch() path is gone.

What this doesn't fix

The server-side streaming issue (stream headers before full serialization) is still needed for very large datasets and very slow phones. ActivityWatch/aw-server-rust#721 covers that. This PR raises the practical timeout cap and makes failures visible until that lands.

Testing

  • Small export: works as before
  • Large export: no longer a silent no-op; shows spinner; shows error if still too slow
  • Error state: dismissible alert appears; button re-enables after failure or timeout

Replaces the axios-based export path with fetch() to fix two stacked
issues that cause large exports (500k+ events) to silently no-op on
Android and desktop:

1. The server serializes all events to a temp file before sending the
   first byte. For 500k events this can exceed the 30 s axios default
   timeout, so the request fails silently with no user feedback.
2. JSON.stringify(response.data, null, 2) re-serializes already-valid
   JSON in the browser JS heap, doubling peak memory use.

The fetch-based path:
- Reads the response as text (no JS parse + re-stringify)
- Uses a 5-minute AbortController timeout instead of 30 s
- Shows a spinner and disables the button while the export runs
- Displays a user-visible error alert on failure (no more silent drop)

The server-side fix (stream headers before full serialization) is still
needed for very large datasets; this raises the practical cap and makes
failures visible until that lands.

Closes ActivityWatch/aw-android#228 (partial)

Git-Session-Id: b1c3
@codecov

codecov Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 53.84615% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.66%. Comparing base (22cb53b) to head (2005bb6).

Files with missing lines Patch % Lines
src/util/export.ts 53.84% 5 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #993      +/-   ##
==========================================
+ Coverage   57.62%   57.66%   +0.03%     
==========================================
  Files          51       51              
  Lines        3219     3231      +12     
  Branches      751      794      +43     
==========================================
+ Hits         1855     1863       +8     
+ Misses       1348     1291      -57     
- Partials       16       77      +61     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@greptile-apps

greptile-apps Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge; the finite timeout resolves the remaining stalled-export state without regressing authentication or concurrent export tracking.

Summary

This PR changes bucket JSON exports to avoid parsing and re-serializing large responses in JavaScript.

  • Uses the authenticated Axios client with blob responses and a finite five-minute timeout.
  • Delegates Android exports to the native URL bridge when available.
  • Tracks overlapping exports and presents progress and failure states.
  • Adds coverage for the Android bridge and export request configuration.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[User starts JSON export] --> B{Android export bridge available?}
  B -->|Yes| C[Delegate URL and filename to native bridge]
  B -->|No| D[Increment in-flight counter]
  D --> E[Authenticated Axios request]
  E --> F[Blob response with five-minute timeout]
  F --> G[Save blob without JSON parsing]
  E -->|Failure or timeout| H[Show dismissible export error]
  G --> I[Decrement in-flight counter]
  H --> I
Loading

Reviews (3) · Last reviewed commit: "fix(export): use a 5-minute axios timeou..."

Comment thread src/views/Buckets.vue Outdated
Comment thread src/views/Buckets.vue Outdated
fetch() dropped the axios Authorization header (Greptile P1) and a single
exporting flag raced concurrent clicks (P2). Use the existing axios client
with responseType blob and timeout 0 so the token is sent, skip
JSON.parse/pretty-print, count in-flight exports, and hand the URL to
Android.exportFromUrl when the WebView bridge is present.

ActivityWatch/aw-android#228

Git-Session-Id: a180614b-5a5a-5d29-84d8-e1c4e076b990
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

CI-green and mergeable — waiting only on a maintainer click.

This PR is ready to merge, but the bot has pull-only access to this repo and can't self-merge — surfacing it here so it isn't lost. The monitoring loop will stop re-flagging it now that this note is posted.

@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

@greptileai review

@TimeToBuildBob TimeToBuildBob changed the title fix(export): use fetch with 5-min timeout for large bucket exports fix(export): stream large bucket exports without JSON.parse Sep 23, 2026
Comment thread src/views/Buckets.vue Outdated
timeout: 0 disabled Axios's timeout, so a stalled export never settled
and export_inflight never decremented. Keep the authenticated blob
request, but cap it at 300s so the spinner and button recover.

ActivityWatch#993

Git-Session-Id: b7e9049b-1299-57f7-8f4a-3c161bf846ab
@TimeToBuildBob

Copy link
Copy Markdown
Contributor Author

@greptileai review

@ErikBjare
ErikBjare merged commit c8c0306 into ActivityWatch:master Sep 23, 2026
9 checks passed
TimeToBuildBob added a commit to TimeToBuildBob/aw-webui that referenced this pull request Sep 23, 2026
timeout: 0 was added for the Tauri fallback path but the existing test
(added in ActivityWatch#993) explicitly verifies no timeout: 0 remains. The server-side
CSV streaming endpoint is fast to start (headers sent before serialization),
so 5 minutes matches the large-event-JSON path and is sufficient.

Git-Session-Id: 70d5
ErikBjare pushed a commit that referenced this pull request Sep 23, 2026
* feat(buckets): use server-side CSV export endpoint

Replace the client-side PapaParse approach with a direct request to the
new server endpoint GET /api/0/buckets/{id}/export/csv.

In a regular browser (including Android WebView), an anchor navigation
is used so the browser handles the download natively — no events ever
enter JS memory and headers arrive immediately, fixing the apparent hung
connection on large buckets.

In Tauri (where <a download> is not supported), the CSV text is fetched
via the API client and saved through the native file dialog.

Removes the papaparse import from this file.

Git-Session-Id: cdc6

* fix(buckets): use 300s timeout for Tauri CSV export, not infinite

timeout: 0 was added for the Tauri fallback path but the existing test
(added in #993) explicitly verifies no timeout: 0 remains. The server-side
CSV streaming endpoint is fast to start (headers sent before serialization),
so 5 minutes matches the large-event-JSON path and is sufficient.

Git-Session-Id: 70d5

* fix(buckets): authenticate CSV export and surface download errors

The browser path used a bare <a> navigation, which drops Authorization
and cannot report 4xx/network failures. Use the same path as JSON
export: Android native URL download, then an authenticated blob GET
with the 5-minute timeout. downloadBlob still handles Tauri.

Git-Session-Id: d11ab6e2-235d-55ab-8e68-e80e65a02e47

* fix(buckets): stream unauthenticated browser CSV downloads

Restore <a> navigation for local installs so the server-side CSV
endpoint can stream. Keep the authenticated blob path for Tauri and
Bearer-token deployments, which cannot attach Authorization to a
navigation download.

Git-Session-Id: 92d2be57-41cd-59e0-8f80-fc2b1a31d6af
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants