polars array reading inconsistent dtypes - #329
Open
TeunHuijben wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix Array(Float64) columns being truncated to ints on SQL reads
Reading an array column back from a
SQLGraphcan silently drop the fractional part of the values, when the leading rows happen to hold whole numbers:No error, no warning, just wrong numbers. Swap the two rows and it works,
graph.nodes[id]["pos"]is correct, and the in-memory backends are fine — only the bulk read on SQL truncates.Array columns are stored as pickled blobs, so
_polars_schema_overridedeliberately leaves them out (declaringArray(Float64, 2)there gives all-None, I tried). The loss happens a step later inunpickle_bytes_columns: once the blobs are unpickled, the column is rebuilt withpl.Series(values)and no dtype, so polars infers it from the leading rows —pl.Series([[50, 50], [1.5, 1.5]])givesList(Int64)and[[50, 50], [1, 1]]._cast_columnsthen casts the already-truncated values to Float64. We knew the declared dtype all along, we just never passed it.Changes
unpickle_bytes_columnstakes an optionaldtypesmapping and builds each column with its declared dtype, falling back to inference, then to object (forMask).SQLGraph._declared_column_dtypes(table, pickled=...), which now backs both_polars_schema_override(natively stored columns) and the new_pickled_column_dtypes(blob columns). Wired into all four read sites.pl.Objectcolumns build directly as object instead of being inferred into a struct and cast back, which drops a raise-per-read on mask columns.Tests
test_array_attr_read_honors_declared_dtype, parametrized over all backends, coveringArray(Float64, 2)andList(Float64)with a whole-numbered first row. Fails onSQLGraphonly before the fix.