Problem
read_csvs (src/ispypsa/data_fetch/csv_read_write.py) is a bare pd.read_csv, so column dtypes are inferred from file contents. Two shapes our data model treats as normal come back with the wrong types:
- Header-only CSV (the "all columns, no rows" convention) → every column is
object, including int/datetime keys.
- A string column that is blank in every row (wildcard / fallback keys such as
timeslice) → the column is float64.
In-process frames keep their dtypes, so this is latent until the CLI round-trips tables through CSV (e.g. dodo.py reading pypsa_friendly_dir back). Any merge on one of these keys can then fail or mis-join, and the failure surfaces as an unrelated-looking pandas dtype error at the merge site rather than at the read.
Example (from #137)
link_timeslice_limits with only fallback rows (blank timeslice) plus a header-only timeslice_snapshots:
ValueError: You are trying to merge on float64 and object columns for key 'timeslice'
raised from _place_named_limits_at_snapshots in src/ispypsa/pypsa_build/links.py. This is the shape the translator emits for zero-capacity corridors / all-default paths in a run with no timeslices. An xfail test in tests/test_model/test_add_links_with_timeslice_limits.py pins it.
Proposed fix
Fix at the read boundary rather than casting at each merge site: add YAML schemas for the pypsa-friendly tables (mirroring src/ispypsa/validation/schemas/ for the input tables) and have the reader apply their declared column types (dtype= / parse_dates= or an immediate coerce) so empty tables and all-blank string columns arrive with the schema's dtypes.
Problem
read_csvs(src/ispypsa/data_fetch/csv_read_write.py) is a barepd.read_csv, so column dtypes are inferred from file contents. Two shapes our data model treats as normal come back with the wrong types:object, including int/datetime keys.timeslice) → the column isfloat64.In-process frames keep their dtypes, so this is latent until the CLI round-trips tables through CSV (e.g.
dodo.pyreadingpypsa_friendly_dirback). Any merge on one of these keys can then fail or mis-join, and the failure surfaces as an unrelated-looking pandas dtype error at the merge site rather than at the read.Example (from #137)
link_timeslice_limitswith only fallback rows (blanktimeslice) plus a header-onlytimeslice_snapshots:raised from
_place_named_limits_at_snapshotsinsrc/ispypsa/pypsa_build/links.py. This is the shape the translator emits for zero-capacity corridors / all-default paths in a run with no timeslices. An xfail test intests/test_model/test_add_links_with_timeslice_limits.pypins it.Proposed fix
Fix at the read boundary rather than casting at each merge site: add YAML schemas for the pypsa-friendly tables (mirroring
src/ispypsa/validation/schemas/for the input tables) and have the reader apply their declared column types (dtype=/parse_dates=or an immediate coerce) so empty tables and all-blank string columns arrive with the schema's dtypes.