git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Testing an updating side input on global window


Hi Lukasz, many thanks for your responses.

I'm actually using them but I think I'm not being able to synchronise the following steps:
1: The side input gets its first value (v1)
2: The main stream gets that side input applied and finds that v1 value
3: The side one gets updated (v2)
4: The main stream gets the side input applied again and finds the v2 value (along with v1 as this is multimap)

Regards

On Tue, May 29, 2018 at 10:57 PM Lukasz Cwik <lcwik@xxxxxxxxxx> wrote:
Your best bet is to use TestStreams[1] as it is used to validate window/triggering behavior. Note that the transform requires special runner based execution and currently only works with the DirectRunner. All examples are marked with the JUnit category "UsesTestStream", for example [2]. 

On Tue, May 29, 2018 at 1:05 PM Carlos Alonso <carlos@xxxxxxxxxxxxx> wrote:
Hi all!!

Basically that's what I'm trying to do. I'm building a pipeline that has a refreshing, multimap, side input (BQ schemas) that then I apply to the main stream of data (records that are ultimately saved to the corresponding BQ table).

My job, although being of streaming nature, runs on the global window, and I want to unit test that the side input refreshes and that the updates are successfully applied.

I'm using scio and I can't seem to simulate that refreshing behaviour. These are the relevant bits of the code: https://gist.github.com/calonso/87d392a65079a66db75a78cb8d80ea98

The way I see understand it, the side collection is refreshed before accessing it so when accessed, it already contains the final (updated) snapshot of the schemas, is that true? In which case, how can I simulate that synchronisation? I'm using processing times as I thought that could be the way to go, but obviously something is wrong there.

Many thanks!!