Re: Question regarding Streaming Resources
On 2018/09/12 20:42:22, Ken Krugler <kkrugler_lists@xxxxxxxxxxxx> wrote:
> Hi Bhaskar,
> I assume you don’t have 1000 streams, but rather one (keyed) stream with 1000 different key values, yes?
> If so, then this one stream is physically partitioned based on the parallelism of the operator following the keyBy(), not per unique key.
> The most common per-key “resource” is the memory required for each key's state, if you’ve got any operations that need to maintain state (accumulators, windows, etc).
> For 1000 unique keys, this should be negligible.
> — Ken
> > On Sep 12, 2018, at 9:55 AM, bhaskar.ebay77@xxxxxxxxx <mailto:bhaskar.ebay77@xxxxxxxxx> wrote:
> > Hi
> > I have created a KeyedStream with state as explained below
> > For example i have created 1000 streams, out of which 50% of streams data is going to come once in 8 hours. Will the resources of these under utilized streams are idle for that duration? Or Flink internal task manager is having some strategy to utilize them for other new streams that are coming?
> > Regards
> > Bhaskar
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
> Custom big data solutions & training
> Flink, Solr, Hadoop, Cascading & Cassan
As per documentation it is showing: https://ci.apache.org/projects/flink/flink-docs-release-1.6/dev/stream/operators/
On DataStream if we apply KeyBy then output is KeyedStream. Once its stream means it should execute in parallel right? There will be 1000 streams each is having Keyed State. What you are saying is the main over head here is only memory. That means Does these 1000 streams are going to be run across 1000 task slots in parallel? These 1000 task slots is the main memory over head? Even 50% of them idle its not harm?