git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [DISCUSS] Flink 1.4 and below STOPS writing to Kinesis after June 12th.


-->

On Wed, Jun 6, 2018 at 1:06 AM, Bowen Li <bowenli86@xxxxxxxxx> wrote:

> Hi,
>
> I think the following email thread might have gone lost.
>
> Dyana brought up the attention that AWS has informed users that KPL
> versions
> < 0.12.6 will *stop working* starting from the 12th of June. Flink 1.4 is
> using KPL 0.12.5 and Flink 1.5 uses 0.12.6, *so Flink 1.4 and below (1.3,
> etc) will be impacted.*
>
> I think we probably should try our best to communicate this out to our
> users in user email alias and all other possible channels, in order not to
> disrupt their production pipeline.
>
> A quick solution we can suggest to users is to package and use
> flink-connector-kinesis in Flink 1.5. Given that the public APIs don't
> change, flink-connector-kinesis in Flink 1.5 should work with Flink 1.4 and
> below, but that needs verification.
>

The connector might work, but it is going to require a little extra
juggling for the users to exclude the 1.5 transitive dependencies.

How about just backporting the changes? We internally use the newer SDK
with a number of other consumer patches on top of 1.4.2 and it works fine.

Normally such changes don't qualify for a patch version, but since it is
going to stop working anyways, there is probably no harm in this instance.

Thomas



> What do you think?
>
> Thanks, Bowen
>
>
> ---------- Forwarded message ----------
> From: Bowen Li <bowenli86@xxxxxxxxx>
> Date: Fri, May 11, 2018 at 10:28 AM
> Subject: Re: KPL in current stable 1.4.2 and below, upcoming problem
> To: dev@xxxxxxxxxxxxxxxx, "Tzu-Li (Gordon) Tai" <tzulitai@xxxxxxxxxx>
>
>
> Thanks, this is a great heads-up!  Flink 1.4 is using KPL 0.12.5, *so Flink
> version 1.4 or below will be effected.*
>
> Kinesis sink is in flink-kinesis-connector. Good news is that whoever is
> using flink-kinesis-connector right now is building it themself, because
> Flink doesn't publish that jar into maven due to licensing issue. So these
> users, like you Dyana,  already have build experience and may try to bump
> KPL themselves.
>
> I think it'll be great if Flink can bump KPL in Flink 1.2/1.3/1.4 and
> release minor versions for them, as an official support. It also requires
> checking backward compatibility. This can be done after releasing 1.5.
> @Gordon may take the final call of how to eventually do it.
>
> Thanks,
> Bowen
>
>
> On Thu, May 10, 2018 at 1:35 AM, Dyana Rose <dyana.rose@xxxxxxxxxxxxx>
> wrote:
>
> > Hello,
> >
> > We've received notification from AWS that the Kinesis Producer Library
> > versions < 0.12.6 will stop working after the 12th of June (assuming the
> > date in the email is in US format...)
> >
> > Flink v1.5.0 has the KPL version at 0.12.6 so it will be fine when it's
> > released. However using the kinesis connector in any previous version
> looks
> > like they'll have an issue.
> >
> > I'm not sure how/if you want to communicate this. We build Flink
> ourselves,
> > so I plan on having a look at any changes done to the Kinesis Sink in
> > v1.5.0 and then bumpimg the KPL version in our fork and rebuilding.
> >
> > Thanks,
> > Dyana
> >
> > below is the email we received (note: we're in eu-west-1):
> > --------
> >
> > Hello,
> >
> >
> >
> > Your action is required: please update clients running Kinesis Producer
> > Library 0.12.5 or older or you will experience a breaking change to your
> > application.
> >
> >
> >
> > We've discovered you have one or more clients writing data to Amazon
> > Kinesis Data Streams running an outdated version of the Kinesis Producer
> > Library. On 6/12 these clients will be impacted if they are not updated
> to
> > Kinesis Producer Library version 0.12.6 or newer. On 06/12 Kinesis Data
> > Streams will install ATS certificates which will prevent these outdated
> > clients from writing to a Kinesis Data Stream. The result of this change
> > will break any producer using KPL 0.12.5 or older.
> >
> >
> > * How do I update clients and applications to use the latest version of
> the
> > Kinesis Producer Library?
> >
> > You will need to ensure producers leveraging the Kinesis Producer Library
> > have upgraded to version 0.12.6 or newer. If you operate older versions
> > your application will break due untrusted SSL certification.
> >
> > Via Maven install Kinesis Producer Library version 0.12.6 or higher [2]
> >
> > After you've configured your clients to use the new version, you're done.
> >
> > * What if I have questions or issues?
> >
> > If you have questions or issues, please contact your AWS Technical
> Account
> > Manager or AWS support and file a support ticket [3].
> >
> > [1] https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-u
> > pgrades.html
> >
> > [2] http://search.maven.org/#artifactdetails|com.amazonaws|amazo
> > n-kinesis-produ...
> > <http://search.maven.org/#artifactdetails%7Ccom.amazonaws%7C
> > amazon-kinesis-producer%7C0.12.6%7Cjar>
> >
> > [3] https://aws.amazon.com/support
> >
> >
> >
> > -          Amazon Kinesis Data Streams Team
> > -----
> >
>