Skip to content

polars array reading inconsistent dtypes - #329

Open
TeunHuijben wants to merge 1 commit into
royerlab:mainfrom
TeunHuijben:fix-pl-array-read-wrong-dtype
Open

polars array reading inconsistent dtypes#329
TeunHuijben wants to merge 1 commit into
royerlab:mainfrom
TeunHuijben:fix-pl-array-read-wrong-dtype

Conversation

@TeunHuijben

Copy link
Copy Markdown
Contributor

Fix Array(Float64) columns being truncated to ints on SQL reads

Reading an array column back from a SQLGraph can silently drop the fractional part of the values, when the leading rows happen to hold whole numbers:

graph = td.graph.SQLGraph(drivername="sqlite", database=":memory:")
graph.add_node_attr_key("pos", pl.Array(pl.Float64, 2))
graph.bulk_add_nodes([{"t": 0, "pos": [50, 50]},     # whole numbers
                      {"t": 1, "pos": [1.5, 1.5]}])  # fractional

graph.node_attrs(attr_keys=["pos"])["pos"].to_list()
# -> [[50.0, 50.0], [1.0, 1.0]]     the 1.5 became 1.0

No error, no warning, just wrong numbers. Swap the two rows and it works, graph.nodes[id]["pos"] is correct, and the in-memory backends are fine — only the bulk read on SQL truncates.

Array columns are stored as pickled blobs, so _polars_schema_override deliberately leaves them out (declaring Array(Float64, 2) there gives all-None, I tried). The loss happens a step later in unpickle_bytes_columns: once the blobs are unpickled, the column is rebuilt with pl.Series(values) and no dtype, so polars infers it from the leading rows — pl.Series([[50, 50], [1.5, 1.5]]) gives List(Int64) and [[50, 50], [1, 1]]. _cast_columns then casts the already-truncated values to Float64. We knew the declared dtype all along, we just never passed it.

Changes

  • unpickle_bytes_columns takes an optional dtypes mapping and builds each column with its declared dtype, falling back to inference, then to object (for Mask).
  • Added SQLGraph._declared_column_dtypes(table, pickled=...), which now backs both _polars_schema_override (natively stored columns) and the new _pickled_column_dtypes (blob columns). Wired into all four read sites.
  • Side benefit: pl.Object columns build directly as object instead of being inferred into a struct and cast back, which drops a raise-per-read on mask columns.

Tests

test_array_attr_read_honors_declared_dtype, parametrized over all backends, covering Array(Float64, 2) and List(Float64) with a whole-numbered first row. Fails on SQLGraph only before the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reading a pl.Array(pl.Float64) column back determines dtypes by first row

1 participant