git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Looking for relevant sources related to connecting Apache Flink and Edgent.


Hi Felipe,

You can define TableSources (for SQL, Table API) that support filter
push-down.
The optimizer will figure out this opportunity and hand filters to a custom
TableSource.

I should add that AFAIK this feature is not used very often (expect some
rough edges) and that the API is likely to change in the future.
But it might be enough for a simple POC.

Best, Fabian

Am Fr., 30. Nov. 2018 um 10:13 Uhr schrieb Felipe Gutierrez <
felipe.o.gutierrez@xxxxxxxxx>:

> I guess this message from 2016 is very related of what I am looking for (
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Execution-Plan-td4290.html).
> I am posting here for future references.
>
> I am going to implement a toy example to visualize this. Do you guys see
> this description as actual of latest Flink source code?
> *--*
> *-- Felipe Gutierrez*
>
> *-- skype: felipe.o.gutierrez*
> *--* *https://felipeogutierrez.blogspot.com
> <https://felipeogutierrez.blogspot.com>*
>
>
> On Thu, Nov 29, 2018 at 12:01 PM Felipe Gutierrez <
> felipe.o.gutierrez@xxxxxxxxx> wrote:
>
>> thanks Kostas for the quick reply,
>>
>> yes. It is related to my previous question.
>>
>> When you said "But if you know what operation to push down" -> This is
>> what I am trying to search on Flink code. I want to know the operation on
>> the fly.
>> The component on Flink that will say to me that there is a filter on the
>> query specified by the user. I want to get this metadata and send a message
>> to my RPi through a Flink connector (I guess this is the way to do) and the
>> data stream will come to Flink already filtered.
>>
>> I intend to start with a simple and naive example. Do you know which
>> component on Flink I can get the operations on the fly that are
>> running inside a query?
>>
>> thanks
>> *--*
>> *-- Felipe Gutierrez*
>>
>> *-- skype: felipe.o.gutierrez*
>> *--* *https://felipeogutierrez.blogspot.com
>> <https://felipeogutierrez.blogspot.com>*
>>
>>
>> On Thu, Nov 29, 2018 at 11:18 AM Kostas Kloudas <
>> k.kloudas@xxxxxxxxxxxxxxxxx> wrote:
>>
>>> Hi again,
>>>
>>> I forgot to say that, unfortunately, I am not familiar with Apache
>>> Edgent, but if you can write your filter in Edgent's programming model,
>>> Then you can push your data from Edgent to a third party storage system
>>> (e.g. Kafka, HDFS, etc) and use Flink's connectors, instead of
>>> having to implement a custom source.
>>>
>>> Cheers,
>>> Kostas
>>>
>>> On Thu, Nov 29, 2018 at 11:08 AM Kostas Kloudas <
>>> k.kloudas@xxxxxxxxxxxxxxxxx> wrote:
>>>
>>>> Hi Felipe,
>>>>
>>>> This seems related to your previous question about a custom scheduler
>>>> that knows which task to run on which machine.
>>>> As Chesnay said, this is a rather involved and laborious task, if you
>>>> want to do it as a general framework.
>>>>
>>>> But if you know what operation to push down, then why not decoupling
>>>> the two and implementing the filtering as a separate job
>>>> running on your Raspberry and a new job which consumes the output of
>>>> the first and does the analytics?
>>>>
>>>> Cheers,
>>>> Kostas
>>>>
>>>> On Thu, Nov 29, 2018 at 10:23 AM Felipe Gutierrez <
>>>> felipe.o.gutierrez@xxxxxxxxx> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am trying to design a little prototype with Flink and Apache Edgent (
>>>>> http://edgent.apache.org/) and I would like some help on the
>>>>> direction for it. I am running Flink at my laptop and Edgent on my
>>>>> Raspberry Pi with a simple filter for a proximity sensor (
>>>>> https://github.com/felipegutierrez/explore-rpi/blob/master/src/main/java/org/sense/edgent/app/UltrasonicEdgentApp.java
>>>>> ).
>>>>>
>>>>> My idea is to push down the filter operator from Flink to the
>>>>> Raspberry Pi which is running Apache Edgent. With this in mind, where do
>>>>> you guys advise me to start?
>>>>>
>>>>> I have some ideas to study...
>>>>> 1 - Try to get the list of operators that Flink is about to execute on
>>>>> the JobManager. source:
>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/internals/job_scheduling.html
>>>>> 2 - Implement a connector to Apache Edgent in order to exchange
>>>>> messages between Flink and Edgent.
>>>>>
>>>>> Do you guys think in another source that is interesting regarding my
>>>>> prototype?
>>>>>
>>>>> Thanks,
>>>>> Felipe
>>>>> *--*
>>>>> *-- Felipe Gutierrez*
>>>>>
>>>>> *-- skype: felipe.o.gutierrez*
>>>>> *--* *https://felipeogutierrez.blogspot.com
>>>>> <https://felipeogutierrez.blogspot.com>*
>>>>>
>>>>