Skip to content

[bugfix] fix invalid sql generated for hive table with multiple partitions - #557

Open
Linxiushen wants to merge 2 commits into
alibaba:masterfrom
Linxiushen:fix-hive-gen-sql-multi-partition
Open

Linxiushen wants to merge 2 commits into
alibaba:masterfrom
Linxiushen:fix-hive-gen-sql-multi-partition

Conversation

@Linxiushen

@Linxiushen Linxiushen commented Sep 21, 2026 •

Copy link
Copy Markdown

Problem

TableInfo.gen_sql() in easy_rec/python/utils/hive_utils.py builds the partition filter by joining one k=v predicate per partition and then joining them with a space:

for k, v in self.partition_kv.items():
    res.append('{}={}'.format(k, v))
part = ' '.join(res)

For a table path with more than one partition, which HiveUtils._construct_table_info explicitly documents (sample_table/dt=2014-11-23/name=a), the generated SQL is

select col1,col2
    from sample_table where dt=2014-11-23 name=a

which is not a valid multi-partition predicate. The loop is clearly written for N partitions; only the separator is wrong.

Fix

Join the predicates with ' and '. For zero or one partition ' '.join and ' and '.join produce the same string, so those paths are unchanged. I checked this with a differential over 9 table paths x 5 limit values: the 10 outputs that differ are exactly the multi-partition cases, and the change in each is the separator only.

Left alone on purpose: partition values are still emitted unquoted (dt=2014-11-23 is an arithmetic expression in SQL). That is a separate issue and changing it would alter behaviour for every single-partition user, so it is not part of this PR.

Tests

Two cases added to easy_rec/python/test/util_test.py, constructing TableInfo directly with an OrderedDict so the assertion is deterministic on both the Python 2 and Python 3 CI pipelines:

  • test_hive_gen_sql_multi_partitions: supplies quoted string literals, fails on master (where dt='2014-11-23' name='a'), and passes with the separator fix.
  • test_hive_gen_sql_single_partition: pins the unchanged single-partition output including the limit clause; passes before and after.

For this test-fixture revision, the exact focused test bodies were run against master and the PR head by AST extraction because TensorFlow is unavailable locally. py_compile, Flake8 on the changed test file, and YAPF --diff passed. Hive integration and the repository-pinned pre-commit hooks were not rerun for this revision.


This fix was developed with AI assistance (Claude); the change and the tests were reviewed and verified locally before submission.

…tions

TableInfo.gen_sql joined the partition predicates with a space, so a table
path with more than one partition, e.g. sample_table/dt=2014-11-23/name=a
as described in HiveUtils._construct_table_info, produced

    where dt=2014-11-23 name=a

which is not valid SQL. Join the predicates with ' and '. Tables with zero
or one partition produce exactly the same SQL as before.

Adds two tests in util_test.py: one for the multi-partition where clause
and one pinning the single-partition output (including the limit clause).
@dajiaohuang

Copy link
Copy Markdown

One additional correctness question, separate from the predicate separator: _construct_table_info documents paths such as sample_table/dt=2014-11-23/name=a, and it passes partition values through as strings. With a synthetic path sample_table/dt=2026-01-02/name=example, the proposed separator change still emits where dt=2026-01-02 and name=example; those unquoted values are parsed as expressions/identifiers rather than string literals. Is the input contract that values must already contain SQL quotes, or should literal quoting be handled separately? The added tests assert the generated text but do not validate execution semantics. No Hive service/data is needed to see the generated string.

@Linxiushen

Copy link
Copy Markdown
Author

Thanks for flagging this. _construct_table_info passes path values through as strings; there is no documented contract requiring callers to prequote them. This PR fixes only the missing conjunction. I updated the multi-partition test to supply quoted SQL string literals so its assertion isolates that fix. Handling bare values properly needs a separate policy for typed, numeric, and already quoted values, including escaping. The checks here are focused SQL-generation checks, not a Hive integration test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants