Good intuition... yarn.scheduler.minimum-allocation-mb is set to 2048 :)
I specified -ytm 2048 and -yjm 2048 and the job started right away, I will also try again later to see if it's not luck. Thanks a lot !
Regarding the version, it is still 0.1, and that I have no clue.... I downloaded 1.5.2 from this link : https://www.apache.org/dyn/closer.lua/flink/flink-1.5.2/flink-1.5.2-bin-scala_2.11.tgz
It should be the official build... It seems to be correct afterall, I see "1a9b648" there too: https://github.com/apache/flink/releases
But I don't know why it's written version 0.1...
De : Gary Yao <gary@xxxxxxxxxxxxxxxxx>
Envoyé : mardi 7 août 2018 19:30
À : Florian Simond
Cc : user@xxxxxxxxxxxxxxxx
Objet : Re: Could not build the program from JAR file.
Thank you for the logs. They look indeed strange but I cannot reproduce this
behavior. From the logs I can see that the ResourceManager is requesting
containers with different resource profiles (2048mb or 1024mb memory):
Requesting new TaskExecutor container with resources <memory:2048, vCores:1>. Number pending requests 1.
Requesting new TaskExecutor container with resources <memory:1024, vCores:1>. Number pending requests 1.
At the moment I am not exactly sure how this happens, and if this is
problematic at all. It would be helpful to know if you are configuring YARN
with a yarn.scheduler.minimum-allocation-mb of 2048.
Here are some other things to try out for troubleshooting:
Can you try raising the the TM and JM memory to 2048mb (-ytm 2048 -yjm 2048)?
You are setting -ytm to 1024, which results in a heap size of only 424mb.
Is the deployment also slow if you are running example/streaming/WordCount.jar?
The version in the log shows up as:
"Version: 0.1, Rev:1a9b648, Date:25.07.2018 @ 17:10:13 GMT".
This does not seem to be an official Flink 1.5.2 distribution. Is there are
reason for that, and can you rule out that there are no changes? Maybe try out
the official build?
On Tue, Aug 7, 2018 at 2:44 PM, Florian Simond <florian.simond@xxxxxxxxxx> wrote: