git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Multiple stream operator watermark handling


Hi,

From top of my head I can imagine two solutions:

1. Override the default behaviour of the operator via for example org.apache.flink.streaming.api.datastream.ConnectedStreams#transform

2. Can you set control stream’s watermark to Watermark#MAX_WATERMARK or maybe Watermark#MAX_WATERMARK - 1 ?

Piotrek

> On 24 May 2018, at 16:07, Elias Levy <fearsome.lucidity@xxxxxxxxx> wrote:
> 
> Is there mechanism for a multiple stream operator to ignore watermarks from one of the streams?
> 
> The use case is a multiple stream operator that consumes a primary stream and a secondary control stream.  The control stream may only receive messages in rare occasion, and possibly never.  The default behavior of the operator is to only emit the lowest of the last watermark received from each input stream.  That means that event time fails to advance if there are no control messages.  
> 
> I also notice that FLIP-17, the Side Input proposal, does not address this issue, either in the Wiki or in the Google Docs.
> 
> Assuming there is no currently prescribed way to handle this, are folks taking care of this by introducing a new Assigner after the multiple input operator to generate watermarks?
> 
>