git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Do I still need hadoop-aws libs when using Flink 1.5 and Presto?


Thanks for pick up my question. I had s3a in the config now I removed it.
I will post a full trace soon, but want to get some questions answered to help me understand this better.

1. Can I use the presto lib with Flink 1.5 without bundled hdp? Can I use this? http://www.apache.org/dyn/closer.lua/flink/flink-1.5.0/flink-1.5.0-bin-scala_2.11.tgz
2. How do I configure presto for endpoints, encryption? The S3A file system needed core-site.yml to configure such things and S3 V4 signature. Do I have to do it for presto?
3. If yes, how to do it? Just add s3.xxx to flink-config? 
    like s3.server-side-encryption-algorithm: AES256
    s3.endpoint: 's3.amazonaws.com' other values for France regions, etc

I will post more logs when I get one. Thanks

On Tue, Jun 5, 2018 at 9:09 AM Aljoscha Krettek <aljoscha@xxxxxxxxxx> wrote:
Hi,

what are you using as the FileSystem scheme? s3 or s3a?

Also, could you also post the full stack trace, please?

Best,
Aljoscha


On 2. Jun 2018, at 07:34, Hao Sun <hasun@xxxxxxxxxxx> wrote:

I am trying to figure out how to use S3 as state storage.

Seems like I only have to do two things:
1. Put flink-s3-fs-presto to the lib
2. Configure
s3.access-key: your-access-key
s3.secret-key: your-secret-key

But I see this exception: ClassNotFoundException: NativeS3FileSystem/S3AFileSystem Not Found


Add it is suggested to add more libs.
So I am confused here, is there a step 3 needed? Isn't the presto jar is all self contained?

Thanks



( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-flink-users/msg09687.html on line 71
Call Stack
#TimeMemoryFunctionLocation
10.0007368696{main}( ).../msg09687.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-flink-users/msg09687.html on line 71
Call Stack
#TimeMemoryFunctionLocation
10.0007368696{main}( ).../msg09687.html:0