git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[DISCUSS] Unified Core API for Streaming and Batch


Hi all,
This post proposes unified core API for Streaming and Batch. 
Currently DataStream and DataSet adopt separated compilation processes, execution tasks
and basic programming models in the runtime layer, which complicates the system implementation. 
We think that batch jobs can be processed in the same way as streaming jobs, thus we can unify
the execution stack of DataSet into that of DataStream.  After the unification the DataSet API will
also be built on top of StreamTransformation, and its basic programming model will be changed
from "UDF on Driver" to "UDF on StreamOperator". Although the DataSet operators will need to
implement the interface StreamOperator instead after the unification, user jobs do not need to change
since DataSet uses the same UDF interfaces as DataStream.

The unification has at least three benefits:
1. The system will be greatly simplified with the same execution stack for both streaming and batch jobs.
2. It is no longer necessary to implement two sets of Driver(s) (operator strategies) for batch, namely chained and non-chained.
3. The unified programming model enables streaming and batch jobs to share the same operator implementation.

The following is the design draft. Any feedback is highly appreciated. .https://docs.google.com/document/d/1G0NUIaaNJvT6CMrNCP6dRXGv88xNhDQqZFrQEuJ0rVU/edit?usp=sharing

Best, 
Haibo