Repository navigation
[bugfix] fix invalid sql generated for hive table with multiple partitions - #557
Linxiushen wants to merge 2 commits into
Conversation
…tions
TableInfo.gen_sql joined the partition predicates with a space, so a table
path with more than one partition, e.g. sample_table/dt=2014-11-23/name=a
as described in HiveUtils._construct_table_info, produced
where dt=2014-11-23 name=a
which is not valid SQL. Join the predicates with ' and '. Tables with zero
or one partition produce exactly the same SQL as before.
Adds two tests in util_test.py: one for the multi-partition where clause
and one pinning the single-partition output (including the limit clause).
|
One additional correctness question, separate from the predicate separator: |
|
Thanks for flagging this. |
Problem
TableInfo.gen_sql()ineasy_rec/python/utils/hive_utils.pybuilds the partition filter by joining onek=vpredicate per partition and then joining them with a space:For a table path with more than one partition, which
HiveUtils._construct_table_infoexplicitly documents (sample_table/dt=2014-11-23/name=a), the generated SQL iswhich is not a valid multi-partition predicate. The loop is clearly written for N partitions; only the separator is wrong.
Fix
Join the predicates with
' and '. For zero or one partition' '.joinand' and '.joinproduce the same string, so those paths are unchanged. I checked this with a differential over 9 table paths x 5 limit values: the 10 outputs that differ are exactly the multi-partition cases, and the change in each is the separator only.Left alone on purpose: partition values are still emitted unquoted (
dt=2014-11-23is an arithmetic expression in SQL). That is a separate issue and changing it would alter behaviour for every single-partition user, so it is not part of this PR.Tests
Two cases added to
easy_rec/python/test/util_test.py, constructingTableInfodirectly with anOrderedDictso the assertion is deterministic on both the Python 2 and Python 3 CI pipelines:test_hive_gen_sql_multi_partitions: supplies quoted string literals, fails onmaster(where dt='2014-11-23' name='a'), and passes with the separator fix.test_hive_gen_sql_single_partition: pins the unchanged single-partition output including thelimitclause; passes before and after.For this test-fixture revision, the exact focused test bodies were run against
masterand the PR head by AST extraction because TensorFlow is unavailable locally.py_compile, Flake8 on the changed test file, and YAPF--diffpassed. Hive integration and the repository-pinned pre-commit hooks were not rerun for this revision.This fix was developed with AI assistance (Claude); the change and the tests were reviewed and verified locally before submission.