fix np scalars becoming blobs in SQLGraph - #327
Conversation
|
was noise, rerunning the benchmarks resolved it :) |
|
|
||
| node_ids.append(node_id) | ||
| insert_rows.append({**node, DEFAULT_ATTR_KEYS.NODE_ID: node_id}) | ||
| insert_rows.append({**node, DEFAULT_ATTR_KEYS.T: time, DEFAULT_ATTR_KEYS.NODE_ID: node_id}) |
There was a problem hiding this comment.
Was this a bug? Why wasn't "t" part of this dict before?
There was a problem hiding this comment.
No, it wasn't a bug. And in hindsight it was not necessary, so I removed it. t is part of **node, so it is added. Because _flatten_attrs_for_write runs right after this, it will make sure t is casted correctly. The reason why we need the time-conversion a few lines above is for the node_id calculation.
| for edge in edges: | ||
| _data_numpy_to_native(edge) |
There was a problem hiding this comment.
Did you remove this because of the _flatten_attrs_for_write change?
There was a problem hiding this comment.
Yes, we call _flatten_attrs_for_write later, which takes care of the checking
| attrs = attrs.copy() | ||
| _data_numpy_to_native(attrs) |
There was a problem hiding this comment.
yes, same as for bulk_add_edges
|
Thanks for the review @JoOkuma, does this look better? |
Fix numpy scalars being stored as BLOBs on SQLGraph insert
Inserting a node whose attribute is a numpy integer scalar corrupts the value on
SQLGraph: it gets written as a raw byte buffer instead of an integer, and reading the column back blows up.sqlite3doesn't knownp.int64, so it falls back to the buffer protocol and stores the raw 8 bytes, even though the column is declaredBIGINT.np.float64andnp.str_survive because they subclass their Python counterparts, which is why this is easy to miss. The in-memory backends handle numpy scalars fine, so code that works there breaks only when you switch to SQL.We already converted numpy scalars in
bulk_add_edgesand_update_table—bulk_add_nodeswas the one write path that didn't. This came up downstream in funtracks, where geff/CSV imports feed numpy scalars straight from the loaded arrays intobulk_add_nodes.Changes
_flatten_attrs_for_write, which all four write paths share, and run it after struct flattening so numpy values nested inside a struct attribute are covered too (they weren't before).tinbulk_add_nodesbefore computing the node id. A narrow numpytoverflowed the id arithmetic silently:np.int32(3) * 1_000_000_000wraps to-1294967296, giving a negative, colliding node id with no error. Same forindices, so the returned ids and thenode_addedpayload stay plain ints.add_overlap/bulk_add_overlaps.isinstance(v, np.generic), which is clearer thannp.isscalar(v) and hasattr(v, "item")and can't touch 0-d arrays.Tests
Three new tests in
test_graph_backends.py, parametrized over all backends, covering plain attrs, struct attrs and the update path, plus one SQL-only test pinning the id overflow.