git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Routing events by key


JB, reshuffle deprecation is discussed mutliple times, most recent one is from May : https://lists.apache.org/thread.html/820064a81c86a6d44f21f0d6c68ea3f46cec151e5e1a0b52eeed3fbf@%3Cdev.beam.apache.org%3E
I filed https://issues.apache.org/jira/browse/BEAM-4372. Summary: It can be used, and perhaps needs a rename. Just mentioned the thread here as a reference, further discussion could be directed to the above dev thread or the jira.

Thanks Robert for more detailed description.

Raghu. 

On Fri, Jul 6, 2018 at 11:11 PM Jean-Baptiste Onofré <jb@xxxxxxxxxxxx> wrote:
Hi Raghu,

AFAIR, Reshuffle is considered as deprecated. Maybe it would be better
to avoid to use it no ?

Regards
JB

On 06/07/2018 18:28, Raghu Angadi wrote:
> I would use Reshuffle()[1] with entity id as the key. It internally does
> a GroupByKey and sets up windowing such that it does not buffer anything.
>
> [1]
> : https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Reshuffle.java#L51 
>
> On Fri, Jul 6, 2018 at 8:01 AM Niels Basjes <Niels@xxxxxxxxx
> <mailto:Niels@xxxxxxxxx>> wrote:
>
>     Hi,
>
>     I have an unbounded stream of change events each of which has the id
>     of the entity that is changed.
>     To avoid the need for locking in the persistence layer that is
>     needed in part of my processing I want to route all events based on
>     this entity id.
>     That way I know for sure that all events around a single entity go
>     through the same instance of my processing sequentially, hence no
>     need for locking or other synchronization regarding this persistence.
>
>     At this point my best guess is that I need to use the GroupByKey but
>     that seems to need a Window. 
>     I think I don't want a window because the stream is unbounded and I
>     want the lowest possible latency (i.e. a Window of 1 second would be
>     ok for this usecase).
>     Also I want to be 100% sure that all events for a specific id go to
>     only a single instance because I do not want any race conditions.
>
>     My simple question is: What does that look like in the Beam Java API?
>
>     --
>     Best regards / Met vriendelijke groeten,
>
>     Niels Basjes
>

--
Jean-Baptiste Onofré
jbonofre@xxxxxxxxxx
http://blog.nanthrax.net
Talend - http://www.talend.com