git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Looking for relevant sources related to connecting Apache Flink and Edgent.


Hi again,

I forgot to say that, unfortunately, I am not familiar with Apache Edgent,
but if you can write your filter in Edgent's programming model,
Then you can push your data from Edgent to a third party storage system
(e.g. Kafka, HDFS, etc) and use Flink's connectors, instead of
having to implement a custom source.

Cheers,
Kostas

On Thu, Nov 29, 2018 at 11:08 AM Kostas Kloudas <k.kloudas@xxxxxxxxxxxxxxxxx>
wrote:

> Hi Felipe,
>
> This seems related to your previous question about a custom scheduler that
> knows which task to run on which machine.
> As Chesnay said, this is a rather involved and laborious task, if you want
> to do it as a general framework.
>
> But if you know what operation to push down, then why not decoupling the
> two and implementing the filtering as a separate job
> running on your Raspberry and a new job which consumes the output of the
> first and does the analytics?
>
> Cheers,
> Kostas
>
> On Thu, Nov 29, 2018 at 10:23 AM Felipe Gutierrez <
> felipe.o.gutierrez@xxxxxxxxx> wrote:
>
>> Hi,
>>
>> I am trying to design a little prototype with Flink and Apache Edgent (
>> http://edgent.apache.org/) and I would like some help on the direction
>> for it. I am running Flink at my laptop and Edgent on my Raspberry Pi with
>> a simple filter for a proximity sensor (
>> https://github.com/felipegutierrez/explore-rpi/blob/master/src/main/java/org/sense/edgent/app/UltrasonicEdgentApp.java
>> ).
>>
>> My idea is to push down the filter operator from Flink to the Raspberry
>> Pi which is running Apache Edgent. With this in mind, where do you guys
>> advise me to start?
>>
>> I have some ideas to study...
>> 1 - Try to get the list of operators that Flink is about to execute on
>> the JobManager. source:
>> https://ci.apache.org/projects/flink/flink-docs-stable/internals/job_scheduling.html
>> 2 - Implement a connector to Apache Edgent in order to exchange messages
>> between Flink and Edgent.
>>
>> Do you guys think in another source that is interesting regarding my
>> prototype?
>>
>> Thanks,
>> Felipe
>> *--*
>> *-- Felipe Gutierrez*
>>
>> *-- skype: felipe.o.gutierrez*
>> *--* *https://felipeogutierrez.blogspot.com
>> <https://felipeogutierrez.blogspot.com>*
>>
>