git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Review Request 67126: HIVE-19326: stats auto gather: incorrect aggregation during UNION queries (may lead to incorrect results)


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67126/
-----------------------------------------------------------

Review request for hive, Ashutosh Chauhan and Sergey Shelukhin.


Bugs: HIVE-19326
    https://issues.apache.org/jira/browse/HIVE-19326


Repository: hive-git


Description
-------

in queries like: INSERT ... SELECT ... UNION ALL SELECT ... 
the stats are only collected for the first select

there are 2 issues fixed - which both resulted in the same result:

* statscollectors have overwritten eachothers result; because the filename was only dependent from the resulting table name
* in case tez.merge.files the 2. task have not been set to collect statistics


Diffs
-----

  itests/src/test/resources/testconfiguration.properties 13c08de3c57fdd7fcd4181814bb8e547c699b9f1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 01a5b4c9c328cb034a613a1539cea2584e122fb4 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 108bb57c4189b720e672eb6f09b1ef05f78448c2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 07991811f92bc7accae2fde23244f424bdd64c6b 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 605bb09caba3b60ad8d51e50dace70757ab80188 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsCollectionContext.java 5c3328c63e8ca37e284e1dc1cdbee5969e185a80 
  ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsPublisher.java 902b37f7874dd5b1afaf8c8bb1259c6f0ddf817f 
  ql/src/test/queries/clientpositive/union_fast_stats.q d69bef3ac083d5a06acda9f47e5d2c1cbe2dfb69 
  ql/src/test/queries/clientpositive/union_rowcounts.q PRE-CREATION 
  ql/src/test/queries/clientpositive/union_stats.q 0e91c23fea475ec95a7fa67433707cb290b277a2 
  ql/src/test/results/clientpositive/llap/multiMapJoin1.q.out f8adcd4ba24f2122f7e7e20770e24a71cfb01a7e 
  ql/src/test/results/clientpositive/llap/union_fast_stats.q.out 4ca5f47a850ba62290c0845eb11a8d0a32780526 
  ql/src/test/results/clientpositive/llap/union_rowcounts.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/union_stats.q.out 5a088f40f5fea41d04a5b6cb6c12bf852a22f097 
  ql/src/test/results/clientpositive/union_stats.q.out 8bd3f44b6e276d6636082673084d66bba3b5c0d3 


Diff: https://reviews.apache.org/r/67126/diff/1/


Testing
-------


Thanks,

Zoltan Haindrich