Skip to content

perf(forwards): backend time-series aggregation, query specialization and server pagination - #779

Open
mutatrum wants to merge 1 commit into
apotdevin:masterfrom
mutatrum:perf/forwards-optimization
Open

mutatrum wants to merge 1 commit into
apotdevin:masterfrom
mutatrum:perf/forwards-optimization

Conversation

@mutatrum

@mutatrum mutatrum commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

My instance of Thunderhub is deployed on a server and manages a routing node with around 160 channels. Thunderhub is unusable on my node, because of the tremendous amounts of data that are pulled to the front-end. For many versions, I patched it locally so that the forwards summary stops at 1 month, the year total crashes my browser. The root cause it repeated requerying of 365 days of forwards, and summing them on the frontend. This takes 28 MB of data, about 14 seconds to get it to the front-end and then another 14 seconds to compute the summary, all the while blocking the tab. The forwards page has the same timings. This is with 1D history alone, the 7D and 1M I can open with great pain, the 2M or 6M are impossible to open. If I leave the dashboard open, it will run out of memory or the tab will get killed by Chrome due to it needing to re-do all the queries and calculations on every forward.

This PR moves all the calculations, summaries and aggregations to the backend, into specialised GraphQL queries. When coupled with #778, a complete reload of the dashboard now takes less that one second and transfers 160kb from the server to the front-end, including the year aggregates. Switching to the 6M view takes about 1.5 seconds. Instead of pulling over >50k forwards, only a limited set and aggregates/time series numbers are sent to the frontend.

The summary of the most important changes:

  • Specialised GraphQL queries by_time and summary
  • Deduplicate channel information in forwards
  • LND gRPC batch size increase. With 55000 forwards, batching in 50's is painful.
  • Remove UUIDv5 and Big.js for forwards. These are expensive for tight loops.

I tried to limit this PR as much as possible, but it's a bit of a monolithic change to get to a usable system for a somewhat busy forwarding node. There are still some further improvements possible:

  • Invoices and payments should also be aggregated on the backend. Currently they are limited to 50 or 100 iirc, so the charts will just not show any data if a node received more than those numbers.
  • The sankey blows up at some point, because it goes over the maximum allowed canvas size. It should only show the routes that are relevant, a single forward a few months ago doesn't warrant an entry on the sankey, or they should be aggregated into an 'the rest' category.
  • Adding a 1 year overview for the forwards. Should be possible. I can add that to this PR if requested.
  • Query deduplication in case multiple widgets want the same data

TLDR: I've been using Thunderhub for many years and this finally scratches my itch. I hope you accept my contribution. There's more to follow if you want.

Proof of forwards:

image

Fixes #590

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Main screen and forwards screen unresponsive due to new year totals of forwards

1 participant