git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Custom Unbounded Sources for 2.4.0 SDK


Ok, so, you should not use native source as it would break the portability.

Instead, just use KafkaIO !

Regards
JB

On 09/06/2018 22:12, Abdul Qadeer wrote:
> Alternate way while using FlinkKafkaConsumer* (:
> Will the Beam Flink runner code need several changes for this?
> 
> On Sat, 9 Jun 2018 at 12:42, Abdul Qadeer <quadeer.leo@xxxxxxxxx
> <mailto:quadeer.leo@xxxxxxxxx>> wrote:
> 
>     I want to use 'FlinkKafkaConsumer' with 'UnboundedFlinkSource' as
>     present in 0.6.0 SDK and it's examples (
>     https://github.com/dataArtisans/flink-dataflow/blob/master/examples/src/main/java/com/dataartisans/flink/dataflow/examples/streaming/KafkaWindowedWordCountExample.java
>     ). 
> 
>     As I understand you are saying this support is not there now,
>     correct? How else could I use a 'FlinkKafkaConsumer' at source
>     level? Is there an alternate you would suggest if not possible?
> 
>     On Sat, 9 Jun 2018 at 07:23, Jean-Baptiste Onofré <jb@xxxxxxxxxxxx
>     <mailto:jb@xxxxxxxxxxxx>> wrote:
> 
>         By the away, if you mean that your custom source is implemented for
>         Flink, it's not supported. I meant the Beam source.
> 
>         On 09/06/2018 09:31, Abdul Qadeer wrote:
>         > Hi!
>         >
>         > I would like to know if there is any way I can use 2.4.0
>         Beam's Source
>         > API for Flink 1.4.0 runner? I have a custom unbounded source
>         implemented
>         > for Flink runner but I can not find the documentation to use
>         it for 2.x
>         > Beam SDK. Looks like it was only supported in 0.x SDK? Any
>         help appreciated.
> 
>         -- 
>         Jean-Baptiste Onofré
>         jbonofre@xxxxxxxxxx <mailto:jbonofre@xxxxxxxxxx>
>         http://blog.nanthrax.net
>         Talend - http://www.talend.com
> 

-- 
Jean-Baptiste Onofré
jbonofre@xxxxxxxxxx
http://blog.nanthrax.net
Talend - http://www.talend.com