git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Testing an updating side input on global window


Your best bet is to use TestStreams[1] as it is used to validate window/triggering behavior. Note that the transform requires special runner based execution and currently only works with the DirectRunner. All examples are marked with the JUnit category "UsesTestStream", for example [2]. 



On Tue, May 29, 2018 at 1:05 PM Carlos Alonso <carlos@xxxxxxxxxxxxx> wrote:
Hi all!!

Basically that's what I'm trying to do. I'm building a pipeline that has a refreshing, multimap, side input (BQ schemas) that then I apply to the main stream of data (records that are ultimately saved to the corresponding BQ table).

My job, although being of streaming nature, runs on the global window, and I want to unit test that the side input refreshes and that the updates are successfully applied.

I'm using scio and I can't seem to simulate that refreshing behaviour. These are the relevant bits of the code: https://gist.github.com/calonso/87d392a65079a66db75a78cb8d80ea98

The way I see understand it, the side collection is refreshed before accessing it so when accessed, it already contains the final (updated) snapshot of the schemas, is that true? In which case, how can I simulate that synchronisation? I'm using processing times as I thought that could be the way to go, but obviously something is wrong there.

Many thanks!!