git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [DISCUSS] Flink 1.6 features


Hi Stephan,

Will  [ https://issues.apache.org/jira/browse/FLINK-5479 ]  (Per-partition
watermarks in FlinkKafkaConsumer should consider idle partitions) be
included in 1.6? As we are seeing more users with this issue on the mailing
lists.

Thanks.
Ben

2018-06-05 5:29 GMT+08:00 Che Lui Shum <shumc@xxxxxxxxxx>:

> Hi Stephan,
>
> Will FLINK-7129 (Support dynamically changing CEP patterns) be included in
> 1.6? There were discussions about possibly including it in 1.6:
> http://mail-archives.apache.org/mod_mbox/flink-user/201803.mbox/%3cCAMq=
> OU7gru2O9JtoWXn1Lc1F7NKcxAyN6A3e58kxctb4b508RQ@xxxxxxxxxxxxxx%3e
>
> Thanks,
> Shirley Shum
>
> [image: Inactive hide details for Stephan Ewen ---06/04/2018 02:21:47
> AM---Hi Flink Community! The release of Apache Flink 1.5 has happ]Stephan
> Ewen ---06/04/2018 02:21:47 AM---Hi Flink Community! The release of Apache
> Flink 1.5 has happened (yay!) - so it is a good time
>
> From: Stephan Ewen <sewen@xxxxxxxxxx>
> To: dev@xxxxxxxxxxxxxxxx, user <user@xxxxxxxxxxxxxxxx>
> Date: 06/04/2018 02:21 AM
> Subject: [DISCUSS] Flink 1.6 features
> ------------------------------
>
>
>
> Hi Flink Community!
>
> The release of Apache Flink 1.5 has happened (yay!) - so it is a good time
> to start talking about what to do for release 1.6.
>
> *== Suggested release timeline ==*
>
> I would propose to release around *end of July* (that is 8-9 weeks from
> now).
>
> The rational behind that: There was a lot of effort in release testing
> automation (end-to-end tests, scripted stress tests) as part of release
> 1.5. You may have noticed the big set of new modules under
> "flink-end-to-end-tests" in the Flink repository. It delayed the 1.5
> release a bit, and needs to continue as part of the coming release cycle,
> but should help make releasing more lightweight from now on.
>
> (Side note: There are also some nightly stress tests that we created and
> run at data Artisans, and where we are looking whether and in which way it
> would make sense to contribute them to Flink.)
>
> *== Features and focus areas ==*
>
> We had a lot of big and heavy features in Flink 1.5, with FLIP-6, the new
> network stack, recovery, SQL joins and client, ... Following something like
> a "tick-tock-model", I would suggest to focus the next release more on
> integrations, tooling, and reducing user friction.
>
> Of course, this does not mean that no other pull request gets reviewed, an
> no other topic will be examined - it is simply meant as a help to
> understand where to expect more activity during the next release cycle.
> Note that these are really the coarse focus areas - don't read this as a
> comprehensive list.
>
> This list is my first suggestion, based on discussions with committers,
> users, and mailing list questions.
>
>   - Support Java 9 and Scala 2.12
>
>   - Smoothen the integration in Container environment, like "Flink as a
> Library", and easier integration with Kubernetes services and other proxies.
>
>   - Polish the remaing parts of the FLIP-6 rewrite
>
>   - Improve state backends with asynchronous timer snapshots, efficient
> timer deletes, state TTL, and broadcast state support in RocksDB.
>
>   - Extends Streaming Sinks:
>      - Bucketing Sink should support S3 properly (compensate for eventual
> consistency), work with Flink's shaded S3 file systems, and efficiently
> support formats that compress/index arcoss individual rows (Parquet, ORC,
> ...)
>      - Support ElasticSearch's new REST API
>
>   - Smoothen State Evolution to support type conversion on snapshot restore
>
>   - Enhance Stream SQL and CEP
>      - Add support for "update by key" Table Sources
>      - Add more table sources and sinks (Kafka, Kinesis, Files, K/V stores)
>      - Expand SQL client
>      - Integrate CEP and SQL, through MATCH_RECOGNIZE clause
>      - Improve CEP Performance of SharedBuffer on RocksDB
>
>
>
>