Skip to content

WIP: Performance improvement for SQLGraph - #322

Draft
yfukai wants to merge 2 commits into
royerlab:mainfrom
yfukai:faster-backend
Draft

WIP: Performance improvement for SQLGraph#322
yfukai wants to merge 2 commits into
royerlab:mainfrom
yfukai:faster-backend

Conversation

@yfukai

@yfukai yfukai commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Human summary

Claude-guided performance improvement for a 20M-node-scale large SQLGraph. Yet to be verified by me.

Copilot summary

This pull request introduces several improvements and new tests for SQL backend handling, benchmarking, and utility code. The main changes include a new interactive latency benchmark, expanded test coverage for SQLGraph time-based optimizations and edge cases, and a migration to Ruff for linting and formatting.

Benchmarking improvements:

  • Added InteractiveLatencyBenchmark in benchmarks/graph_mutations.py to measure per-frame query latency with many time points, specifically targeting scenarios with thousands of frames and small per-frame slices.
  • Introduced TrackletSubgraphBenchmark in benchmarks/tracklet_nodes.py to benchmark assign_tracklet_ids and materializing a single tracklet as a subgraph, including performance of filtering by unindexed attributes.

SQL backend test coverage:

  • Added comprehensive tests in test_graph_backends.py for SQLGraph, covering: bulk updates with mixed types, time filter optimizations, custom indices, SQLite PRAGMA settings, index redundancy, node id handling, time point enumeration, update signal payloads, and chunk sizing. [1] [2]

Linting and formatting:

  • Migrated from isort and black to ruff for linting, import sorting, and formatting in .pre-commit-config.yaml, with notes on avoiding conflicts between formatters.

Utility code improvements:

  • Optimized unpickle_bytes_columns in _dataframe.py to avoid unnecessary type narrowing and per-element dispatch, improving performance and clarity when unpickling binary columns from the database. [1] [2]
  • Removed unused import of polars.selectors from _dataframe.py.

These changes improve performance benchmarking, ensure SQL backend correctness in a wide range of scenarios, and streamline code quality enforcement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant