git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 'mvn verify' fails at flink-hadoop-fs


That flat-out disables all tests in the module, even those that could run on Windows.

We commonly add an OS check to respective tests that skip the tests, with an "Assume.assumeTrue(os!=windows)" statement in a "@BeforeClass" method.

On 10.07.2018 21:00, NEKRASSOV, ALEXEI wrote:
I added lines below to flink-hadoop-fs/pom.xml, and that allowed me to turn off the tests that were failing for me.
Do we want to add this change to master?
If so, do I need to document this new switch somewhere?

(
the build then hang for me at flink-runtime, but that's a different issue
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.969 sec - in org.apache.flink.runtime.taskmanager.TaskManagerRegistrationTest
)

	<build>
		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-surefire-plugin</artifactId>
				<configuration>
					<skipTests>${skipHdfsTests}</skipTests>
				</configuration>
			</plugin>
		</plugins>
	</build>

-----Original Message-----
From: Chesnay Schepler [mailto:chesnay@xxxxxxxxxx]
Sent: Tuesday, July 10, 2018 10:36 AM
To: dev@xxxxxxxxxxxxxxxx; NEKRASSOV, ALEXEI <an4828@xxxxxxx>
Subject: Re: 'mvn verify' fails at flink-hadoop-fs

There's currently no workaround except going in and manually disabling them.

On 10.07.2018 16:32, Chesnay Schepler wrote:
Generally, any test that uses HDFS will fail on Windows. We've
disabled most of them, but some slip through from time to time.

Note that we do not provide any guarantees for all tests passing on
Windows.

On 10.07.2018 16:28, NEKRASSOV, ALEXEI wrote:
I'm running 'mvn clean verify' on Windows with no Hadoop libraries
installed, and the build fails (see below).
What's the solution? Is there a switch to skip Hadoop-related tests?
Or I need to install Hadoop libraries?

Thanks,
Alex


Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.726
sec <<< FAILURE! - in
org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest
org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest  Time elapsed:
1.726 sec  <<< ERROR!
java.lang.UnsatisfiedLinkError:
org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
          at
org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
          at
org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)
          at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996)
          at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:484)
          at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:293)
          at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
          at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:891)
          at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
          at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:503)
          at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:559)
          at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
          at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:708)
          at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1358)
          at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:996)
          at
org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:867)
          at
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702)
          at
org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:374)
          at
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
          at
org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest.createHDFS(HdfsBeha
viorTest.java:65)

Running org.apache.flink.runtime.fs.hdfs.HdfsKindTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.081
sec - in org.apache.flink.runtime.fs.hdfs.HdfsKindTest
Running
org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017
sec - in
org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest

Results :

Tests in error:
    HdfsBehaviorTest.createHDFS:65 ▒ UnsatisfiedLink
org.apache.hadoop.io.nativeio...

Tests run: 24, Failures: 0, Errors: 1, Skipped: 1

[INFO]
---------------------------------------------------------------------
---
[INFO] Reactor Summary:
[INFO]
[INFO] force-shading ...................................... SUCCESS [
2.335 s] [INFO] flink ..............................................
SUCCESS [
29.794 s]
[INFO] flink-annotations .................................. SUCCESS [
2.198 s] [INFO] flink-shaded-hadoop ................................
SUCCESS [  0.226 s] [INFO] flink-shaded-hadoop2
............................... SUCCESS [
11.015 s]
[INFO] flink-shaded-hadoop2-uber .......................... SUCCESS [
16.343 s]
[INFO] flink-shaded-yarn-tests ............................ SUCCESS [
13.653 s]
[INFO] flink-shaded-curator ............................... SUCCESS [
1.386 s] [INFO] flink-test-utils-parent ............................
SUCCESS [  0.191 s] [INFO] flink-test-utils-junit
............................. SUCCESS [  3.318 s] [INFO]
flink-metrics ...................................... SUCCESS [  0.212
s] [INFO] flink-metrics-core .................................
SUCCESS [  3.502 s] [INFO] flink-core
......................................... SUCCESS
[01:30 min]
[INFO] flink-java ......................................... SUCCESS
[01:31 min]
[INFO] flink-queryable-state .............................. SUCCESS [
0.186 s] [INFO] flink-queryable-state-client-java ..................
SUCCESS [  4.099 s] [INFO] flink-filesystems
.................................. SUCCESS [  0.198 s] [INFO]
flink-hadoop-fs .................................... FAILURE [  8.672
s]