git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Looking for relevant sources related to connecting Apache Flink and Edgent.


thanks Kostas for the quick reply,

yes. It is related to my previous question.

When you said "But if you know what operation to push down" -> This is what
I am trying to search on Flink code. I want to know the operation on the
fly.
The component on Flink that will say to me that there is a filter on the
query specified by the user. I want to get this metadata and send a message
to my RPi through a Flink connector (I guess this is the way to do) and the
data stream will come to Flink already filtered.

I intend to start with a simple and naive example. Do you know which
component on Flink I can get the operations on the fly that are
running inside a query?

thanks
*--*
*-- Felipe Gutierrez*

*-- skype: felipe.o.gutierrez*
*--* *https://felipeogutierrez.blogspot.com
<https://felipeogutierrez.blogspot.com>*


On Thu, Nov 29, 2018 at 11:18 AM Kostas Kloudas <k.kloudas@xxxxxxxxxxxxxxxxx>
wrote:

> Hi again,
>
> I forgot to say that, unfortunately, I am not familiar with Apache Edgent,
> but if you can write your filter in Edgent's programming model,
> Then you can push your data from Edgent to a third party storage system
> (e.g. Kafka, HDFS, etc) and use Flink's connectors, instead of
> having to implement a custom source.
>
> Cheers,
> Kostas
>
> On Thu, Nov 29, 2018 at 11:08 AM Kostas Kloudas <
> k.kloudas@xxxxxxxxxxxxxxxxx> wrote:
>
>> Hi Felipe,
>>
>> This seems related to your previous question about a custom scheduler
>> that knows which task to run on which machine.
>> As Chesnay said, this is a rather involved and laborious task, if you
>> want to do it as a general framework.
>>
>> But if you know what operation to push down, then why not decoupling the
>> two and implementing the filtering as a separate job
>> running on your Raspberry and a new job which consumes the output of the
>> first and does the analytics?
>>
>> Cheers,
>> Kostas
>>
>> On Thu, Nov 29, 2018 at 10:23 AM Felipe Gutierrez <
>> felipe.o.gutierrez@xxxxxxxxx> wrote:
>>
>>> Hi,
>>>
>>> I am trying to design a little prototype with Flink and Apache Edgent (
>>> http://edgent.apache.org/) and I would like some help on the direction
>>> for it. I am running Flink at my laptop and Edgent on my Raspberry Pi with
>>> a simple filter for a proximity sensor (
>>> https://github.com/felipegutierrez/explore-rpi/blob/master/src/main/java/org/sense/edgent/app/UltrasonicEdgentApp.java
>>> ).
>>>
>>> My idea is to push down the filter operator from Flink to the Raspberry
>>> Pi which is running Apache Edgent. With this in mind, where do you guys
>>> advise me to start?
>>>
>>> I have some ideas to study...
>>> 1 - Try to get the list of operators that Flink is about to execute on
>>> the JobManager. source:
>>> https://ci.apache.org/projects/flink/flink-docs-stable/internals/job_scheduling.html
>>> 2 - Implement a connector to Apache Edgent in order to exchange messages
>>> between Flink and Edgent.
>>>
>>> Do you guys think in another source that is interesting regarding my
>>> prototype?
>>>
>>> Thanks,
>>> Felipe
>>> *--*
>>> *-- Felipe Gutierrez*
>>>
>>> *-- skype: felipe.o.gutierrez*
>>> *--* *https://felipeogutierrez.blogspot.com
>>> <https://felipeogutierrez.blogspot.com>*
>>>
>>