WIP: Performance improvement for SQLGraph - #322
Draft
yfukai wants to merge 2 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Human summary
Claude-guided performance improvement for a 20M-node-scale large SQLGraph. Yet to be verified by me.
Copilot summary
This pull request introduces several improvements and new tests for SQL backend handling, benchmarking, and utility code. The main changes include a new interactive latency benchmark, expanded test coverage for SQLGraph time-based optimizations and edge cases, and a migration to Ruff for linting and formatting.
Benchmarking improvements:
InteractiveLatencyBenchmarkinbenchmarks/graph_mutations.pyto measure per-frame query latency with many time points, specifically targeting scenarios with thousands of frames and small per-frame slices.TrackletSubgraphBenchmarkinbenchmarks/tracklet_nodes.pyto benchmarkassign_tracklet_idsand materializing a single tracklet as a subgraph, including performance of filtering by unindexed attributes.SQL backend test coverage:
test_graph_backends.pyfor SQLGraph, covering: bulk updates with mixed types, time filter optimizations, custom indices, SQLite PRAGMA settings, index redundancy, node id handling, time point enumeration, update signal payloads, and chunk sizing. [1] [2]Linting and formatting:
isortandblacktorufffor linting, import sorting, and formatting in.pre-commit-config.yaml, with notes on avoiding conflicts between formatters.Utility code improvements:
unpickle_bytes_columnsin_dataframe.pyto avoid unnecessary type narrowing and per-element dispatch, improving performance and clarity when unpickling binary columns from the database. [1] [2]polars.selectorsfrom_dataframe.py.These changes improve performance benchmarking, ensure SQL backend correctness in a wide range of scenarios, and streamline code quality enforcement.