git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

SQL Filter Pushdowns in Apache Beam SQL


Hi,

We are currently playing with Apache Beam’s SQL extension on top of Flink. One of the features that we were interested is the SQL Predicate Pushdown feature that Spark provides. Does Beam support that?

For eg:
I have an unbounded dataset that I want to join with some static reference data stored in a database. Will beam perform the logic of figuring out all the unique keys in the window and push it down to the jdbc source or will it bring all the data from the jdbc source into memory and then perform the join?

Thanks,
Harsh
--
Regards,
Harshvardhan