Skip to content

fix np scalars becoming blobs in SQLGraph - #327

Open
TeunHuijben wants to merge 2 commits into
royerlab:mainfrom
TeunHuijben:fix-numpy-in-sql
Open

fix np scalars becoming blobs in SQLGraph#327
TeunHuijben wants to merge 2 commits into
royerlab:mainfrom
TeunHuijben:fix-numpy-in-sql

Conversation

@TeunHuijben

Copy link
Copy Markdown
Contributor

Fix numpy scalars being stored as BLOBs on SQLGraph insert

Inserting a node whose attribute is a numpy integer scalar corrupts the value on SQLGraph: it gets written as a raw byte buffer instead of an integer, and reading the column back blows up.

graph = td.graph.SQLGraph(drivername="sqlite", database=":memory:")
graph.add_node_attr_key("val", dtype=pl.Int64, default_value=-1)
graph.add_node({"t": 0, "val": np.int64(7)})
graph.node_attrs(attr_keys=["val"])
# ComputeError: could not append value: b"\x07\x00\x00\x00\x00\x00\x00\x00" of type: binary

sqlite3 doesn't know np.int64, so it falls back to the buffer protocol and stores the raw 8 bytes, even though the column is declared BIGINT. np.float64 and np.str_ survive because they subclass their Python counterparts, which is why this is easy to miss. The in-memory backends handle numpy scalars fine, so code that works there breaks only when you switch to SQL.

We already converted numpy scalars in bulk_add_edges and _update_tablebulk_add_nodes was the one write path that didn't. This came up downstream in funtracks, where geff/CSV imports feed numpy scalars straight from the loaded arrays into bulk_add_nodes.

Changes

  • Moved the conversion into _flatten_attrs_for_write, which all four write paths share, and run it after struct flattening so numpy values nested inside a struct attribute are covered too (they weren't before).
  • Coerce t in bulk_add_nodes before computing the node id. A narrow numpy t overflowed the id arithmetic silently: np.int32(3) * 1_000_000_000 wraps to -1294967296, giving a negative, colliding node id with no error. Same for indices, so the returned ids and the node_added payload stay plain ints.
  • Coerce ids in add_overlap / bulk_add_overlaps.
  • Narrowed the check to isinstance(v, np.generic), which is clearer than np.isscalar(v) and hasattr(v, "item") and can't touch 0-d arrays.

Tests

Three new tests in test_graph_backends.py, parametrized over all backends, covering plain attrs, struct attrs and the update path, plus one SQL-only test pinning the id overflow.

@TeunHuijben TeunHuijben linked an issue Jul 30, 2026 that may be closed by this pull request
@TeunHuijben

TeunHuijben commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

Benchmark shows some huge regressions though, I will investigate...

was noise, rerunning the benchmarks resolved it :)

@TeunHuijben
TeunHuijben requested a review from JoOkuma July 30, 2026 17:34
Comment thread src/tracksdata/graph/_sql_graph.py Outdated

node_ids.append(node_id)
insert_rows.append({**node, DEFAULT_ATTR_KEYS.NODE_ID: node_id})
insert_rows.append({**node, DEFAULT_ATTR_KEYS.T: time, DEFAULT_ATTR_KEYS.NODE_ID: node_id})

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was this a bug? Why wasn't "t" part of this dict before?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it wasn't a bug. And in hindsight it was not necessary, so I removed it. t is part of **node, so it is added. Because _flatten_attrs_for_write runs right after this, it will make sure t is casted correctly. The reason why we need the time-conversion a few lines above is for the node_id calculation.

Comment on lines -1166 to -1167
for edge in edges:
_data_numpy_to_native(edge)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you remove this because of the _flatten_attrs_for_write change?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we call _flatten_attrs_for_write later, which takes care of the checking

Comment on lines -2037 to -2038
attrs = attrs.copy()
_data_numpy_to_native(attrs)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, same as for bulk_add_edges

@TeunHuijben

Copy link
Copy Markdown
Contributor Author

Thanks for the review @JoOkuma, does this look better?

@TeunHuijben
TeunHuijben requested a review from JoOkuma July 30, 2026 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SQLGraph doesn't handle np.int well in .add_node

2 participants