fix(manifest_splits): refuse case-colliding column names instead of renaming them - #685
Conversation
|
| else: | ||
| source_columns.append(column_name) | ||
| nested_fields_remaining = child_count or 0 | ||
| if len({column.casefold() for column in source_columns}) != len(source_columns): |
There was a problem hiding this comment.
Valid Unicode columns block splits
split_manifest rejects a Parquet file with columns such as Ä and ä. Python’s casefold() treats these names as equal, but DuckDB keeps them distinct and would not rename either one. The same check also rejects valid group_columns pairs. Compare names the way DuckDB does so callers can split these manifests.
kstonekuan
left a comment
There was a problem hiding this comment.
Thanks, the fix is right and the test is good. One change: _reject_ambiguous_column_names is a copy of the inline guard in src/hflow/manifest_deduplication.py:130-143. Keep one copy. Define the function once, call it from both split_manifest and the deduplication path, and delete the inline loop.
Move _reject_ambiguous_column_names into _manifest_parquet_schema and call it from split_manifest and deduplicate_manifest. Use DuckDB's ASCII identifier fold so Unicode columns such as Ä/ä stay distinct.
|
@kstonekuan yeah good catch, pulled the guard into one shared helper and both split and dedup call it now. also switched the collision check to duckdb's ascii fold so Ä and ä don't count as a clash. |
kstonekuan
left a comment
There was a problem hiding this comment.
LGTM, merging. Thanks.
Summary
split_manifestnow refuses a source Parquet file whose top-level column names collide ignoring case, raisingValueError("manifest column names must be distinct ignoring case").ManifestSplitSettingsalso rejectsgroup_columnsthat differ only by case (e.g.("group", "GROUP")).Why
DuckDB silently renames case-colliding columns when it opens the file (
METADATAbecomesMETADATA_1), so the partition files written bysplit_manifestcarried a different schema than the input, contradicting "preserving every row and column". A case-colliding group column also failed with a misleading "manifest is missing columns" error.deduplicate_manifestalready guards against this by reading the original names fromparquet_schema(?)before the view is created. This applies the same check tosplit_manifestand matches the error message. The check runs before the output directory is created, so nothing is published on refusal.Fixes #683
Validation
uv run ruff check --fix,uv run ruff format,uv run ty check: cleanuv run pytest -q -n 4: 2507 passed, 8 skippedtest_case_colliding_source_columns_are_refused_not_renamedand the extendedtest_policy_rejects_ambiguous_relationships_and_fractionsChecklist
uv run ruff check --fix,uv run ruff format, anduv run ty check.