git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Aggregator State in Keyed Windowed Stream


Hi Ning,

Answer you question: 

And why is rich functions not allowed here?

If you need access to the state API, you can consider using ProcessWindowFunction[1], which allows you to use ProcessWindowFunction.

Thanks, vino.


Ning Shi <ningshi2@xxxxxxxxx> 于2018年9月10日周一 上午11:28写道:
Since the aggregate() function on a keyed window stream does not allow
using rich functions, I can only use an AggregateFunction. Is the
accumulator state of the AggregateFunction backed by RocksDB and
persisted in checkpoints if I use the RocksDB backend. My job looks
like the following,

sourceStream.keyBy(…)
    .timeWindow(Time.seconds(…))
    .trigger(CountTrigger.of(…))
    .aggregate(new MyAggFunc());

Since the stream has high cardinality keys, each window could have
millions of them. If the accumulator state is not backed by RocksDB,
there might be a lot of data stored on the heap.

And why is rich functions not allowed here?

Thanks,


Ning