git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Dynamic output - Python


Hi Eila,

Python textio currently does not support dynamic writes. So, to achieve this, you'll have to implement a custom DoFn that writes data to dynamic destinations. 

Thanks,
Cham

On Mon, Mar 12, 2018 at 6:26 AM OrielResearch Eila Arich-Landkof <eila@xxxxxxxxxxxxxxxxx> wrote:
Hello all,

I would like to print each PCollection element to a different folder on gs://.../
the folder name is not available in advance, but it is available on the pCollection data (or can be calculated prior to printing).
My question are:
- Is it possible to use the folder path as a parameter to the beam.io.WriteToText?
- Can you refer me to a code example where dynamic output (or similar) is being used in Python?
Thanks,
Eila


( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-beam-users/msg02042.html on line 70
Call Stack
#TimeMemoryFunctionLocation
10.0006364552{main}( ).../msg02042.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-beam-users/msg02042.html on line 70
Call Stack
#TimeMemoryFunctionLocation
10.0006364552{main}( ).../msg02042.html:0