git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Trigerring Savepoint for the Flink Job


Hi Anil,

I am actually one of the engineers maintaining Uber's AthenaX open-source platform. As for now, AthenaX is still running on Flink 1.3.2 so it might be weird to follow the 1.4.2 release document. (We are working on upgrading to latest 1.5 release)

For your question regarding savepoints, AthenaX does not support savepoint natively at this moment. But we have a separated API WatchdogPolicy that you can customized. It supports monitoring / management APIs which will be called periodically. You can utilize that to trigger your daily savepoints. In term of how to do savepoint, I think the REST API [1] might be a good starting point in AthenaX case because we launch the job in detach mode.

Please let me know if this is helpful.

Thanks,
Rong

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/rest_api.html#available-requests

On Thu, May 31, 2018 at 1:01 PM, Anil <anilsingh.jsr@xxxxxxxxx> wrote:
I am using Flink 1.4.2. I have forker Uber's AthenaX  project
<https://github.com/uber/AthenaX>  .

The Flink jobs are deployed in Yarn cluster. I needed to save the Savepoint
for all the jobs everyday.

ClusterClient
<https://github.com/apache/flink/blob/release-1.4.2/flink-clients/src/main/java/org/apache/flink/client/program/ClusterClient.java#L672
gave an implementation for saving savepoint using Flink ID.
YarnClusterClient
<https://github.com/apache/flink/blob/release-1.4.2/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterClient.java
is an implementation of ClusteClient.

Initial though was to use YarnClusterClient instance with Flink Id (I save
this when the Flink Job is deployed to Yarn cluster) to trigger savepoint.
So I created an instance of YarnClusterClient once and saved it so that I
could use it anytime in the application. But this doesn't seems to work. It
doesn't seems that it can cancel or trigger savepoint even with valid Flink
ID. When I try to cancel a valid Flink Job it throws and error for invalid
id.

I would appreciate if someone could help me out here.





--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/