Re: withPartitioner() vs calling partitionCustom() beforehand
logically the resulting plans should be identical, meaning that they both
will use the custom partitioner to create the partitions and then co group
Physically, the latter plan adds an additional partition operator before
the coGroup operator. You can see this is you call env.getExecutionPlan()
and then use Flink's plan visualizer . The partition operator adds
another task which instantiates another thread. Consequently,
coGroup(b).where(...).equalTo(...).withPartitioner(...) should be slightly
On Sun, Dec 2, 2018 at 1:35 PM Giannis Evagorou <gevagorou@xxxxxxx> wrote:
> Hi all,
> I have a question regarding partitioning.
> Does calling the withPartitioner() method on a coGroup operation has the
> same effect as performing partitionCustom on both datasets beforehand?
> 1. a.coGroup(b).where(…).equalTo(…).withPartitioner(…).with(…)
> equivalent to:
> 1. DataSet a = aa.partitionCustom(…)
> 2. DataSet b = bb.partitionCustom(…)
> 3. a.coGroup(b).where(…).equalTo(…).with(…)
> Do both snippets perform the same low-level physical partitioning?
> Thank you,