[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Aggregator State in Keyed Windowed Stream

In addition : 

ProcessWindowFunction extends AbstractRichFunction, through getRuntimeContext,you can access keyed state API.

vino yang <yanghua1127@xxxxxxxxx> 于2018年9月10日周一 下午3:19写道:
Hi Ning,

Answer you question: 

And why is rich functions not allowed here?

If you need access to the state API, you can consider using ProcessWindowFunction[1], which allows you to use ProcessWindowFunction.

Thanks, vino.

Ning Shi <ningshi2@xxxxxxxxx> 于2018年9月10日周一 上午11:28写道:
Since the aggregate() function on a keyed window stream does not allow
using rich functions, I can only use an AggregateFunction. Is the
accumulator state of the AggregateFunction backed by RocksDB and
persisted in checkpoints if I use the RocksDB backend. My job looks
like the following,

    .aggregate(new MyAggFunc());

Since the stream has high cardinality keys, each window could have
millions of them. If the accumulator state is not backed by RocksDB,
there might be a lot of data stored on the heap.

And why is rich functions not allowed here?