[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Ship compiled code with broadcast stream ?


This is certainly possible. What you can do is use a BroadcastProcessFunction where you receive the rule code on the broadcast side.

You probably cannot send newly compiled objects this way but what you can do is either send a reference to some compiled jars and load them with the URLClassloader or send the actual String code and invoke the java compiler from your function.

Hope that helps :)


Maxim Parkachov <lazy.gopher@xxxxxxxxx> ezt írta (időpont: 2018. okt. 9., K, 9:38):
Hi everyone,

I have a job with event stream and control stream delivering rules for event transformation. Rules are broadcasted and used in flatMat-like coProcessFunction. Rules are defined in custom JSON format. Amount of rules and complexity rises significantly with every new feature.

What I would like is to ship compiled (serialized ?) code instead of JSON rules to control stream and use this compiled classes directly without additional transformation. This will allow to get more robust testing and will allow to implement much more complex rules. But I'm struggling to understand how to achieve this.

Did someone implement system like this ? Is this possible at all ?

Any help is greatly appreciated,