[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Can JDBCSinkFunction support exectly once?

Thanks for help.
I find that many sink dont't support stream exectly-once. I need use exectly-once Hbase sink and Mysql sink in my work. 
I try to contribute to Hbase Sink firstly
Best Regards,

Fabian Hueske <fhueske@xxxxxxxxx> 于2018年11月21日周三 下午6:34写道:

JDBCSinkFunction is a simple wrapper around the JDBCOutputFormat (the DataSet / Batch API output interface).
Dominik is right, that JDBCSinkFunction does not support exactly-once output.

It is not strictly required that an exactly-once sink implements TwoPhaseCommitFunction.
TPCF is a convenience interface that internally use the CheckpointListener interface. It makes it easier to implement exactly-once sinks but is not the only way to do it.

Implementing an exactly-once JDBC sink by extending TwoPhaseCommitFunction would require to be able to recover (and commit) an open transaction after a task was restarted.
Not sure if the JDBC interface supports this.

Best, Fabian

Am Mi., 21. Nov. 2018 um 11:27 Uhr schrieb Dominik Wosiński <wossyn@xxxxxxxxx>:

As far as I know, the function needs to implement the TwoPhaseCommitFunction and not the CheckpointListener. JDBCSinkFunction does not implement the two-phase commit, so currently it does not support exactly once.

Best Regards,

śr., 21 lis 2018 o 11:07 Jocean shi <jocean.shi@xxxxxxxxx> napisał(a):
Can JDBCSinkFunction support exectly once? Is it that The JDBCSinkFunction dont't implement CheckpointListener meaning JDBCSinkFunction dont't support exectly once?