git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Old job resurrected during HA failover


Hi Elias,

If a job is explicitly canceled, its jobgraph node on ZK will be deleted. 
However, it is worth noting here that Flink enables a background thread to asynchronously delete the jobGraph node, 
so there may be cases where it cannot be deleted. 
On the other hand, the jobgraph node on ZK is the only basis for the JM leader to restore the job. 
There may be an unexpected recovery or an old job resurrection.

Thanks, vino.

2018-08-01 23:13 GMT+08:00 Elias Levy <fearsome.lucidity@xxxxxxxxx>:
For the second time in as many months we've had an old job resurrected during HA failover in a 1.4.2 standalone cluster.  Failover was initiated when the leading JM lost its connection to ZK.  I opened FLINK-10011 with the details.

We are using S3 with the Presto adapter as our distributed store.  After we cleaned up the cluster by shutting down the two jobs started after failover and starting a new job from the last known good checkpoint from the single job running in the cluster before failover, the HA recovery directory looks as follows:

3cmd ls s3://bucket/flink/cluster_1/recovery/
 DIR s3://bucket/flink/cluster_1/recovery/some_job/}}
2018-07-31 17:33 35553 s3://bucket/flink/cluster_1/recovery/completedCheckpoint12e06bef01c5
2018-07-31 17:34 35553 s3://bucket/flink/cluster_1/recovery/completedCheckpoint187e0d2ae7cb
2018-07-31 17:32 35553 s3://bucket/flink/cluster_1/recovery/completedCheckpoint22fc8ca46f02
2018-06-12 20:01 284626 s3://bucket/flink/cluster_1/recovery/submittedJobGraph7f627a661cec
2018-07-30 23:01 285257 s3://bucket/flink/cluster_1/recovery/submittedJobGraphf3767780c00c

submittedJobGraph7f627a661cec appears to be job 2a4eff355aef849c5ca37dbac04f2ff1, the long running job that failed during the ZK failover

submittedJobGraphf3767780c00c appears to be job d77948df92813a68ea6dfd6783f40e7e, the job we started restoring from a checkpoint after shutting down the duplicate jobs

Should submittedJobGraph7f627a661cec exist in the recovery directory if 2a4eff355aef849c5ca37dbac04f2ff1 is no longer running?