git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Job manager logs for previous YARN attempts


Hi,

I am looking into the cause YARN starts new application attempt on Flink 1.5.2. The challenge is getting the logs for the first attempt. After checking YARN I discovered that in the first attempt and the second one application manager (job manager) gets assigned the same container id (is this expected ?)  In this case logs from the first attempt are overwritten? I found that setKeepContainersAcrossApplicationAttempts is enabled here here

The second challenge is understanding if the job will be restored into new application attempts or new application attempt will just have flink running without any job?


Regards,
Pawel

First attempt:

pawel_bartoszek@ip-10-4-X-X ~]$ yarn container -list  appattempt_1538570922803_0020_000001
18/10/08 10:16:16 INFO client.RMProxy: Connecting to ResourceManager at ip-10-4-X-X.eu-west-1.compute.internal/10.4.108.26:8032
Total number of containers :1
                  Container-Id           Start Time          Finish Time                State                 Host    Node Http Address                             LOG-URL
container_1538570922803_0020_02_000001 Mon Oct 08 09:47:17 +0000 2018                  N/A              RUNNING ip-10-4-X-X.eu-west-1.compute.internal:8041 http://ip-10-4-X-X.eu-west-1.compute.internal:8042 http://ip-10-4-X-X.eu-west-1.compute.internal:8042/node/containerlogs/container_1538570922803_0020_02_000001/pawel_bartoszek

Second attempt:
[pawel_bartoszek@ip-10-4-X-X ~]$ yarn container -list  appattempt_1538570922803_0020_000002
18/10/08 10:16:37 INFO client.RMProxy: Connecting to ResourceManager at ip-10-4-X-X.eu-west-1.compute.internal/10.4.X.X:8032
Total number of containers :1
                  Container-Id           Start Time          Finish Time                State                 Host    Node Http Address                             LOG-URL
container_1538570922803_0020_02_000001 Mon Oct 08 09:47:17 +0000 2018                  N/A              RUNNING ip-10-4-X-X.eu-west-1.compute.internal:8041 http://ip-10-4-X-X.eu-west-1.compute.internal:8042 http://ip-10-4-X-X.eu-west-1.compute.internal:8042/node/containerlogs/container_1538570922803_0020_02_000001/pawel_bartoszek