[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Reset kafka offets to latest on restart

Hi Vishal,

AFAIK, the current behavior of kafka source will always use checkpoint state as the start position, ignoring other configuration.
A workaround solution I can come up with is to set a new uuid to your kafka source and restore your job with `allowNonRestoreState`.
Therefore, you can use the way that Rong provided to set the initial start position.

cc. Gordon who know more about the details of kafka source.

Tony Wei

Rong Rong <walterddr@xxxxxxxxx> 於 2018年11月22日 週四 上午8:23寫道:
Hi Vishal,

You can probably try using similar offset configuration as a service consumer.
Maybe this will be useful to look at [1]


On Wed, Nov 21, 2018 at 1:32 PM Jamie Grier <jgrier@xxxxxxxx> wrote:
Hi Vishal,

No, there is no way to do this currently.

On Wed, Nov 21, 2018 at 10:22 AM Vishal Santoshi <vishal.santoshi@xxxxxxxxx> wrote:
Any one ?

On Tue, Nov 20, 2018 at 12:48 PM Vishal Santoshi <vishal.santoshi@xxxxxxxxx> wrote:
Is it possible to have checkpointing but reset the kafka offsets to  latest on restart on failure ?