So it is now normal that it takes around 5 minutes to start processing ? The job is reading from kafka and writing back into another kafka topic. When I start the job, it takes roughly 5 minutes before I get something in the output topic.
I see a lot of
2018-08-07 12:20:34,672 INFO org.apache.flink.yarn.YarnResourceManager - Received new container: XXX - Remaining pending container requests: 0 2018-08-07 12:20:34,672 INFO org.apache.flink.yarn.YarnResourceManager - Returning excess container XXX.
I see a lot of those lines during the first five minutes.
I'm not sure I need to have a static set of TMs, but as we have a limited set of nodes, and several jobs, it could be harder to make sure they do not interfere with each other...
De : Gary Yao <gary@xxxxxxxxxxxxxxxxx>
Envoyé : mardi 7 août 2018 12:27
À : Florian Simond
Cc : vino yang; user@xxxxxxxxxxxxxxxx
Objet : Re: Could not build the program from JAR file.
You can find more information about the re-worked deployment model here:
TaskManagers are started and shut down according to the slot requirements of
the jobs. It is possible to return to the old behavior by setting
in flink-conf.yaml. However, this mode is deprecated and will be removed soon.
Can you explain why you need to have a static set of TMs?
On Tue, Aug 7, 2018 at 12:07 PM, Florian Simond <florian.simond@xxxxxxxxxx> wrote: