[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: BroadcastStream vs Broadcasted DataStream

Hi Pieter-Jan,

The second variant stores the elements of the broadcasted stream in operator (thus non-keyed) state.

On the differences: 

The Broadcast stream is not a keyed stream, so you are not in a keyed context, thus you have no access to keyed state.
Given this, and assuming that you are implementing a CoProcessFunction, then your function should also implement the 
ListCheckpointed interface and store the broadcasted data as ListState.

From a systems perspective, in case of rescaling, then your function should take care of removing duplicated manually
as in the case of scaling down for example, the state of 2 nodes may end up on one. In addition, given that elements are
re-distributed randomly, you may even end up with some elements missing one some nodes. This would be more evident 
in case of scaling up. In this case, nodes will be missing data.

From an API perspective, assuming that the other stream is keyed and it has state, accessing it in order to do some computation
would be impossible.


On Oct 9, 2018, at 2:52 PM, Pieter-Jan Van Aeken <pieterjan.vanaeken@xxxxxxxxxxx> wrote:


I am not sure I fully understand the differences between doing something like

dataStreamX.connect(dataStreamY.broadcast()).process(new CoProcessFunction{})

and this

dataStreamX.connect(dataStreamY.broadcast(descriptor).process(new BroadcastProcessFunction)

Couldn't I manage the state exactly the same way using the first option? The only difference I could find was in read/write permissions for different states. But other than that, the difference in possibilities escapes me and I was hoping someone here could clarify.

Kind regards,



Pieter-Jan Van Aeken

Consultant - Data Engineer
(M) +32 (0) 474 06 64 48


Rue Emile Francqui, 4

1435 Mont-Saint-Guibert

(T) +32 10 75 02 00

 Be green, keep it on the screen