git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: writing to maprfs?


Hello Akanksha,

Thank you for reporting this! It looks that website misses the details how to use “org.apache.beam.sdk.io.hdfs” with different FSs, so I created a Jira to improve this:

WBR,
Alexey


On 24 May 2018, at 10:36, Akanksha Sharma B <akanksha.b.sharma@xxxxxxxxxxxx> wrote:

Hi,

Answering my own question 😊

writing to maprfs worked after I added following property to core-site.xml:-

    <property>
        <name>fs.maprfs.impl</name>
        <value>com.mapr.fs.MapRFileSystem</value>
    </property>


Regards,
Akanksha


From: Akanksha Sharma B
Sent: Thursday, May 24, 2018 9:24:25 AM
To: user@xxxxxxxxxxxxxxx
Subject: writing to maprfs?
 

Hi All,

I have a pipeline; it writes to hdfs, using "org.apache.beam.sdk.io.hdfs" package.
I was hoping that it will work with maprfs as well. However I have been debugging for some days, with no success.

I do not provide hdfsConfiguration from command line, and instead use the configuration file /opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/core-site.xml (please find it attached with the mail)
However I get the stack trace ("No FileSystem for scheme: maprfs"). Please find the detailed trace attached. I made sure that I have/opt/mapr/lib/maprfs-5.2.2-mapr.jar on classpath. I use spark submit to run the pipeline.

If you have already tried writing to maprfs from beam, ccould you please help?

Regards,
Akanksha