git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Aggregator State in Keyed Windowed Stream


Hi Ning,

> Back to my first question, is the accumulator state backed by RocksDB state backend? If so, I don’t need to use rich function for the aggregate function.

the answer is yes, it is backed by state backend (should be RocksDB if you configure it), 
you can trace it through these method calls:

sourceStream.keyBy(…)
    .timeWindow(Time.seconds(…))
    .trigger(CountTrigger.of(…))
gives you WindowedStream,
WindowedStream.aggregate(new MyAggFunc()) creates:
new WindowOperator(windowStateDescriptor = new AggregatingStateDescriptor()), 
inside WindowOperator: 
WindowOperator.open() uses configured backend to create windowState,
WindowOperator.processElement() uses windowState which is AggregatingState.

Cheers,
Andrey

> On 10 Sep 2018, at 13:39, Ning Shi <ningshi2@xxxxxxxxx> wrote:
> 
> Hi Vino,
> 
>> If you need access to the state API, you can consider using ProcessWindowFunction[1], which allows you to use ProcessWindowFunction.
> 
> I was hoping that I could use the aggregate function to do incremental aggregation. My understanding is that ProcessWindowFunction either has to loop through all records or be combined with an aggregate function to do incremental aggregation.
> 
> Back to my first question, is the accumulator state backed by RocksDB state backend? If so, I don’t need to use rich function for the aggregate function.
> 
> Thanks,
> 
> Ning