Skip to content

fix(manifest_splits): refuse case-colliding column names instead of renaming them - #685

Merged
kstonekuan merged 2 commits into
Hebbian-Robotics:mainfrom
cestercian:fix-split-manifest-case-collision
Oct 4, 2026
Merged

kstonekuan merged 2 commits into
Hebbian-Robotics:mainfrom
cestercian:fix-split-manifest-case-collision

Conversation

@cestercian

Copy link
Copy Markdown
Contributor

Summary

split_manifest now refuses a source Parquet file whose top-level column names collide ignoring case, raising ValueError("manifest column names must be distinct ignoring case"). ManifestSplitSettings also rejects group_columns that differ only by case (e.g. ("group", "GROUP")).

Why

DuckDB silently renames case-colliding columns when it opens the file (METADATA becomes METADATA_1), so the partition files written by split_manifest carried a different schema than the input, contradicting "preserving every row and column". A case-colliding group column also failed with a misleading "manifest is missing columns" error.

deduplicate_manifest already guards against this by reading the original names from parquet_schema(?) before the view is created. This applies the same check to split_manifest and matches the error message. The check runs before the output directory is created, so nothing is published on refusal.

Fixes #683

Validation

  • uv run ruff check --fix, uv run ruff format, uv run ty check: clean
  • uv run pytest -q -n 4: 2507 passed, 8 skipped
  • New tests fail without the change: test_case_colliding_source_columns_are_refused_not_renamed and the extended test_policy_rejects_ambiguous_relationships_and_fractions

Checklist

  • I added or updated outcome-focused tests for changed business logic.
  • I updated documentation for changed behavior, flags, formats, or requirements. (No docs describe this; behaviour now matches the existing docstring contract.)
  • I ran uv run ruff check --fix, uv run ruff format, and uv run ty check.
  • I ran the relevant pytest suite.
  • I did not add recordings, generated media, credentials, private URLs, or runtime artifacts.
  • I preserved stored-data compatibility or documented an explicit version change.

@greptile-apps

greptile-apps Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

[Medium risk] Changes how manifest column name validation works.

The PR is not yet safe to merge because valid Unicode group columns still block splitting.

Findings

  1. P1 Valid Unicode columns block splits ▶
Summary

Manifest splitting now refuses Parquet columns whose names differ only by case, and split settings reject group names that differ only by case. This keeps DuckDB from silently renaming columns and changing the schema in split files.

  • Checks original Parquet column names before opening a DuckDB view.
  • Rejects case-colliding group names when settings are created.
  • Reuses the shared source-name check in deduplication and tests that refused splits create no output directory.

Reviews (2) · Last reviewed commit: "refactor(manifest): share parquet column..."

Comment thread src/hflow/manifest_splits.py Outdated
else:
source_columns.append(column_name)
nested_fields_remaining = child_count or 0
if len({column.casefold() for column in source_columns}) != len(source_columns):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Valid Unicode columns block splits

split_manifest rejects a Parquet file with columns such as Ä and ä. Python’s casefold() treats these names as equal, but DuckDB keeps them distinct and would not rename either one. The same check also rejects valid group_columns pairs. Compare names the way DuckDB does so callers can split these manifests.

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the fix is right and the test is good. One change: _reject_ambiguous_column_names is a copy of the inline guard in src/hflow/manifest_deduplication.py:130-143. Keep one copy. Define the function once, call it from both split_manifest and the deduplication path, and delete the inline loop.

Move _reject_ambiguous_column_names into _manifest_parquet_schema and call
it from split_manifest and deduplicate_manifest. Use DuckDB's ASCII
identifier fold so Unicode columns such as Ä/ä stay distinct.
@cestercian

Copy link
Copy Markdown
Contributor Author

@kstonekuan yeah good catch, pulled the guard into one shared helper and both split and dedup call it now. also switched the collision check to duckdb's ascii fold so Ä and ä don't count as a clash.

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, merging. Thanks.

@kstonekuan
kstonekuan merged commit 04fa15c into Hebbian-Robotics:main Oct 4, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: split_manifest silently corrupts output schema when source parquet contains case-colliding column names

2 participants