git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Java based AWS IO connector


Beam works with S3 out of the box, you can provide s3://... paths to anything that works with files. Don't remember if it's already available in 2.4 or only starting 2.5 - does it currently not work for you?

On Tue, May 29, 2018, 2:26 PM S. Sahayaraj <ssahayaraj@xxxxxxxxx> wrote:

Hello All,

                The data source for our Beam pipleline is in S3 bucket, Is there any built-in I/O Connector available with Java samples? If so, can you please guide me how to integrate with them?.

 

                I am using Bean SDK for Java version 2.4.0 and Spark runner in clustered deployment.

 

<dependency>

              <groupId>org.apache.beam</groupId>

              <artifactId>beam-sdks-java-core</artifactId>

              <version>2.4.0</version>

            </dependency>

 

Cheers,

S. Sahayaraj