I have a pipeline; it writes to hdfs, using "org.apache.beam.sdk.io.hdfs" package.
I was hoping that it will work with maprfs as well. However I have been debugging for some days, with no success.
I do not provide hdfsConfiguration from command line, and instead use the configuration file /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/core-site.xml (please find it attached with the mail)
However I get the stack trace ("No FileSystem for scheme: maprfs"). Please find the detailed trace attached. I made sure that I have/opt/mapr/lib/maprfs-5.2.2-mapr.jar on classpath. I use spark submit to run the pipeline.
If you have already tried writing to maprfs from beam, ccould you please help?