git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Review Request 67023: HIVE-18117: Add a new Test Driver "TestErasureCodingHDFSCliDriver" that can be used to run tests over hdfs directories that employ Erasure Coding.



> On May 10, 2018, noon, Sahil Takiar wrote:
> > itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java
> > Lines 674 (patched)
> > <https://reviews.apache.org/r/67023/diff/1/?file=2018287#file2018287line674>
> >
> >     is this necessary since you set the cluster type to mr above?
> 
> Andrew Sherman wrote:
>     Ha good question. Yes it is necessary as setClusterType() does not always set the cluster type :-( - it allows the cluster type to overridden with -Dclustermode=xxx
> 
> Sahil Takiar wrote:
>     interesting, should we handle other cluster types like Spark or MR too?
> 
> Andrew Sherman wrote:
>     looks like it does already

then is the tez specific code necessary?


- Sahil


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67023/#review202831
-----------------------------------------------------------


On May 14, 2018, 9:44 p.m., Andrew Sherman wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67023/
> -----------------------------------------------------------
> 
> (Updated May 14, 2018, 9:44 p.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> TestErasureCodingHDFSCliDriver uses a test-only CommandProcessor "ErasureProcessor"
> which allows .q files to contain Erasure Coding commands similar to those provided
> by the hdfs ec command
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html.
> The Erasure Coding functionality is exposed through a new shim "HdfsFileErasureCodingPolicy".
> At this stage there are two .q files:
> erasure_commnds.q (a simple test to show ERASURE commands can run on local fs via
> TestCliDriver or on hdfs via TestErasureCodingHDFSCliDriver), and
> erasure_simple.q (which does some trivial queries to demonstrate basic functionality).
> More tests will come in future commits.
> 
> 
> Diffs
> -----
> 
>   itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestErasureCodingHDFSCliDriver.java PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties cf6d19a5937c3f4a82e4ffe09201af8a79da2e3d 
>   itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 1814f0fa190e0ebcf327aebcdaf6f9967a5fd14f 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 16571b3ff3288dfb99fbca570452592cc1650f9a 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/CommandProcessorFactory.java 3d47991b603c94c8da2106e67192c8513ef783a7 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/ErasureProcessor.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/processors/HiveCommand.java 56c7516ecfaf2421b0f3d3a188d05f38715b25b2 
>   ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 89129f99fe63f0384aff965ad665770d11e9af04 
>   ql/src/test/org/apache/hadoop/hive/ql/processors/TestCommandProcessorFactory.java de43c2866f64e2ed5c74eab450de28f1a79248dc 
>   ql/src/test/queries/clientpositive/erasure_commands.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q PRE-CREATION 
>   ql/src/test/results/clientpositive/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_commands.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out PRE-CREATION 
>   shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java ec06a88dc21346473bec6589c703167d50e3b367 
>   shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java b89081761780bf1f305d0196bb94bb0b54f7184f 
>   testutils/ptest2/conf/deployed/master-mr2.properties 7edc307f85744d60d322ad8087164625677fc230 
> 
> 
> Diff: https://reviews.apache.org/r/67023/diff/3/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>