Thank you. Flatten.iterables gives the answer to my problem, Great stuff and promising!!. I have now PCollection<ABC> which has more than 4000 datasets (ie 4000 ABC objects in PCollection), that will be executed by ParDo.of(new ExecFn()).
The computing environment here is on Spark cluster which has 8 workers, able to see workers, DAG visualization details on Spark admin UI. But, precisely I would like to visualize the parallel computation on ABC by ExecFn(). Is there any available tool or
app or 3rd party components that helps to figure out parallelism happening on the pipeline? Please suggest.
From: Eugene Kirpichov [mailto:kirpichov@xxxxxxxxxx]
Sent: Thursday, June 7, 2018 10:32 PM
Subject: Re: [EXTERNAL] - Re: Create PCollection<ABC> from List<ABC>in DoFn<>
Also, remember that a DoFn can return multiple results:
And if you have a DoFn<X, List<ABC>> you can follow this with Flatten.iterables to turn the output
PCollection<List<ABC>> into a PCollection<ABC>. In some cases you may want to follow this with a Reshuffle so
that the outputs from a single X get distributed among multiple machines.
If you have a function which given X returns a List<ABC>, you can use FlatMapElements transform
on PCollection<X> to get a PCollection<ABC>.
In case if we could return List<ABC> from DoFn<> then we could use the code as suggested in section 3.1.2 and mentioned by you below., but the return type of DoFn<> is always PCollection<>
in where I could not have the list of ABC objects which further will be fed as input for parallel computation. Is there any possibility to convert List<ABC> to PCollection<ABC> in DoFn<> itself? OR can DoFn<> return List<ABC> objects?
From: Robert Bradshaw [mailto:robertwb@xxxxxxxxxx]
Sent: Wednesday, June 6, 2018 9:40 PM
Subject: [EXTERNAL] - Re: Create PCollection<ABC> from List<ABC>in DoFn<>
You can use the Create transform to do this, e.g.
List<ABC> inMemoryObjects = ...
PCollection<ABC> pcollectionOfObject = p.apply(Create.of(inMemoryObjects));
result = pcollectionOfObject.apply(ParDo.of(SomeDoFn...));
I have created a java class which extends DoFn<>, there are list of objects of type ABC (List<ABC>) populated in processElement(ProcessContext c) at runtime and
would like to generate respective PCollection<ABC> from List<ABC> so that the subsequent transformation can do parallel execution on each ABC object in PCollection<ABC>. How do we create PCollection from in-memory object created in DoFn<>? OR How do we get
pipeline object being in DoFn<>? OR is there any SDK guidelines to refer?