git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [VOTE] Release 1.6.0, release candidate #4


+1

- verified compilation, tests
- verified checksum and gpg files
- verified sbt templates (g8, quickstart) - run assemblies on local cluster

- I could not execute the nightly-tests.sh though. The tests that were
failing most often are:
    - test_streaming_file_sink.sh
    - test_streaming_elasticsearch.sh

Those are connectors though and it might be only tests flakiness so I
think it should not block the release.

On 08/08/18 16:36, Chesnay Schepler wrote:
> I did not use the tools/list_deps.py script as I wasn't aware that it
> existed.
>
> Even if I were I wouldn't have used it and in fact would advocate for
> removing it.
> It manually parses and constructs dependency information which is
> utterly unnecessary as maven already provides this functionality, with
> the added bonus of also accounting for dependencyManagement and
> transitive dependencies which we obviously have to take into account.
>
> I used this one-liner instead:
> |mvn dependency:list | ||grep| |":.*:.*:.*"| || ||grep| |-||v| |-e
> ||"Finished at"| |-e ||"Some problems"| || ||cut| |-d] -f2- | ||sed|
> |'s/:[a-z]*$//g'| || ||sort| |-u
>
> |which I have documented here:
> https://cwiki.apache.org/confluence/display/FLINK/Dependencies
>
> On 08.08.2018 15:06, Aljoscha Krettek wrote:
>> +1
>>
>> - verified checksum and gpg files
>> - verified LICENSE and NOTICE: NOTICE didn't change from 1.5, LICENSE
>> had one unnecessary part removed
>>
>> Side comment: I'm not sure whether the "Verify that the LICENSE and
>> NOTICE file is correct for the binary and source releases" part is
>> valid anymore because we only have one LICENSE and NOTICE file. also
>> "The LICENSE and NOTICE files in flink-dist/src/main/flink-bin refer
>> to the binary distribution and mention all of Flink's Maven
>> dependencies as well" can be dropped because we don't have them anymore.
>>
>> I came to the same conclusion on dependencies. I used
>> tools/list_deps.py and diff'ed the output for 1.5 and 1.6, that's
>> probably what Chesnay also did ... :-)
>>
>>> On 8. Aug 2018, at 14:43, Chesnay Schepler <chesnay@xxxxxxxxxx> wrote:
>>>
>>> +1
>>>
>>> - verified source release contains no binaries
>>> - verified correct versions in source release
>>> - verified compilation, tests and E2E-tests pass (on travis)
>>> - verified checksum and gpg files
>>>
>>> New dependencies (excluding dependencies where we simply depend on a
>>> different version now):
>>>     Apache licensed:
>>>         io.confluent:common-utils:jar:3.3.1
>>>         io.confluent:kafka-schema-registry-client:jar:3.3.1
>>>         io.prometheus:simpleclient_pushgateway:jar:0.3.0
>>>         various Apache Nifi dependencies
>>>         various Apache Parquet dependencies
>>>         various ElasticSearch dependencies
>>>     CDDL:
>>>         javax.ws.rs:javax.ws.rs-api:jar:2.1
>>>     Bouncycastle (MIT-like):
>>>         org.bouncycastle:bcpkix-jdk15on:jar:1.59
>>>         org.bouncycastle:bcprov-jdk15on:jar:1.59
>>>     MIT:
>>>         org.projectlombok:lombok:jar:1.16.20
>>>
>>> On 08.08.2018 13:28, Till Rohrmann wrote:
>>>> Thanks for reporting these problems Chesnay. The usage string in
>>>> `standalone-job.sh` is out dated and should be updated. The same
>>>> applies to
>>>> the typo.
>>>>
>>>> When calling `standalone-job.sh start --job-classname foobar.Job`
>>>> please
>>>> make sure that the user code jar is contained in the classpath (e.g.
>>>> putting the jar in the lib directory). Documenting this behaviour
>>>> is part
>>>> of the pending issue FLINK-10001.
>>>>
>>>> We should fix all of these issues. They are, however, no release
>>>> blockers.
>>>>
>>>> Cheers,
>>>> Till
>>>>
>>>> On Wed, Aug 8, 2018 at 11:31 AM Chesnay Schepler
>>>> <chesnay@xxxxxxxxxx> wrote:
>>>>
>>>>> I found some issues with the standalone-job.sh script.
>>>>>
>>>>> I ran "./bin/standalone-job.sh start" as described by the usage
>>>>> string.
>>>>>
>>>>>      2018-08-08 09:22:34,385 ERROR
>>>>>      org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -
>>>>>      Could not parse command line arguments [--configDir,
>>>>>      /home/zento/svn/flink-1.6.0/flink-1.6.0/conf].
>>>>>      org.apache.flink.runtime.entrypoint.FlinkParseException:
>>>>> Failed to
>>>>>      parse the command line arguments.
>>>>>               at
>>>>>
>>>>> org.apache.flink.runtime.entrypoint.parser.CommandLineParser.parse(CommandLineParser.java:52)
>>>>>
>>>>>               at
>>>>>
>>>>> org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.main(StandaloneJobClusterEntryPoint.java:143)
>>>>>
>>>>>      Caused by: org.apache.commons.cli.MissingOptionException:
>>>>> Missing
>>>>>      required option: j
>>>>>               at
>>>>>
>>>>> org.apache.commons.cli.DefaultParser.checkRequiredOptions(DefaultParser.java:199)
>>>>>
>>>>>               at
>>>>>     
>>>>> org.apache.commons.cli.DefaultParser.parse(DefaultParser.java:130)
>>>>>               at
>>>>>     
>>>>> org.apache.commons.cli.DefaultParser.parse(DefaultParser.java:81)
>>>>>               at
>>>>>
>>>>> org.apache.flink.runtime.entrypoint.parser.CommandLineParser.parse(CommandLineParser.java:50)
>>>>>
>>>>>              ... 1 more
>>>>>
>>>>> The script should fail earlier if no jar is provided, with a better
>>>>> error message.
>>>>> It is also undocumented, and the usage instructions don't appear
>>>>> correct.
>>>>>
>>>>> Passing a jar with the -j option leads to a ClassNotFoundException:
>>>>> "./bin/standalone-job.sh start -j examples/streaming/WordCount.jar"
>>>>>
>>>>>      2018-08-08 09:26:30,562 ERROR
>>>>>      org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -
>>>>>      Cluster initialization failed.
>>>>>      java.lang.reflect.UndeclaredThrowableException
>>>>>               at
>>>>>
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1854)
>>>>>
>>>>>               at
>>>>>
>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>>
>>>>>               at
>>>>>
>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:189)
>>>>>
>>>>>               at
>>>>>
>>>>> org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.main(StandaloneJobClusterEntryPoint.java:158)
>>>>>
>>>>>      Caused by: org.apache.flink.util.FlinkException: Could not
>>>>> load the
>>>>>      provied entrypoint class.
>>>>>               at
>>>>>
>>>>> org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.createPackagedProgram(StandaloneJobClusterEntryPoint.java:92)
>>>>>
>>>>>               at
>>>>>
>>>>> org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.retrieveJobGraph(StandaloneJobClusterEntryPoint.java:75)
>>>>>
>>>>>               at
>>>>>
>>>>> org.apache.flink.runtime.entrypoint.JobClusterEntrypoint.createDispatcher(JobClusterEntrypoint.java:107)
>>>>>
>>>>>               at
>>>>>
>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startClusterComponents(ClusterEntrypoint.java:353)
>>>>>
>>>>>               at
>>>>>
>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:232)
>>>>>
>>>>>               at
>>>>>
>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:190)
>>>>>
>>>>>               at
>>>>> java.security.AccessController.doPrivileged(Native Method)
>>>>>               at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>               at
>>>>>
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>>>>>
>>>>>               ... 3 more
>>>>>      Caused by: java.lang.ClassNotFoundException:
>>>>>      examples/streaming/WordCount.jar
>>>>>               at
>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>>>>               at
>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>>>>               at
>>>>>      sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
>>>>>               at
>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>>>>               at
>>>>>
>>>>> org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.createPackagedProgram(StandaloneJobClusterEntryPoint.java:89)
>>>>>
>>>>>               ... 11 more
>>>>>
>>>>> So this seems to not work at all, but maybe I'm using it wrong?
>>>>>
>>>>> (There's also typo in "Could not load the provied entrypoint class")
>>>>>
>>>>> On 08.08.2018 10:33, Piotr Nowojski wrote:
>>>>>> +1 from my side
>>>>>>
>>>>>> I’ve spent some time playing around with various examples (batching,
>>>>> streaming and SQL) on EMR 6 nodes cluster with yarn deployment, with
>>>>> different configuration options (number of task
>>>>> managers/memory/Flip6/credit base flow control/metrics) and
>>>>> everything
>>>>> looks now fine (after fixing
>>>>> https://issues.apache.org/jira/browse/FLINK-9969 <
>>>>> https://issues.apache.org/jira/browse/FLINK-9969> ).
>>>>>> Piotrek
>>>>>>
>>>>>>> On 7 Aug 2018, at 17:17, Till Rohrmann <trohrmann@xxxxxxxxxx>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi everyone,
>>>>>>> Please review and vote on the release candidate #4 for the version
>>>>> 1.6.0,
>>>>>>> as follows:
>>>>>>> [ ] +1, Approve the release
>>>>>>> [ ] -1, Do not approve the release (please provide specific
>>>>>>> comments)
>>>>>>>
>>>>>>>
>>>>>>> The complete staging area is available for your review, which
>>>>>>> includes:
>>>>>>> * JIRA release notes [1],
>>>>>>> * the official Apache source release and binary convenience
>>>>>>> releases to
>>>>> be
>>>>>>> deployed to dist.apache.org [2], which are signed with the key with
>>>>>>> fingerprint 1F302569A96CFFD5 [3],
>>>>>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>>>>>> * source code tag "release-1.6.0-rc4" [5],
>>>>>>> * website pull request listing the new release and adding
>>>>>>> announcement
>>>>> blog
>>>>>>> post [6].
>>>>>>>
>>>>>>> Please use this document for coordinating testing efforts: [7]
>>>>>>>
>>>>>>> The vote will be shortened since we only adde a minor fix on top
>>>>>>> of the
>>>>> RC
>>>>>>> 3. It will close on Wednesday 6:30pm CET. It is adopted by majority
>>>>>>> approval, with at least 3 PMC affirmative votes.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Your friendly Release Manager
>>>>>>>
>>>>>>> [1]
>>>>>>>
>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12342760
>>>>>
>>>>>>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.6.0/
>>>>>>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>>>>>>> [4]
>>>>> https://repository.apache.org/content/repositories/orgapacheflink-1178
>>>>>
>>>>>>> [5] https://github.com/apache/flink/tree/release-1.6.0-rc4
>>>>>>> [6] https://github.com/apache/flink-web/pull/117
>>>>>>> [7]
>>>>>>>
>>>>> https://docs.google.com/document/d/1upBFZQ7tbaSkYvDiLqfUFXKg8Xxs-lVheEfb66e4jpo/edit?usp=sharing
>>>>>
>>>>>>> Pro-tip: you can create a settings.xml file with these contents:
>>>>>>>
>>>>>>> <settings>
>>>>>>> <activeProfiles>
>>>>>>>    <activeProfile>flink-1.6.0</activeProfile>
>>>>>>> </activeProfiles>
>>>>>>> <profiles>
>>>>>>>    <profile>
>>>>>>>      <id>flink-1.6.0</id>
>>>>>>>      <repositories>
>>>>>>>        <repository>
>>>>>>>          <id>flink-1.6.0</id>
>>>>>>>          <url>
>>>>>>>
>>>>>>> https://repository.apache.org/content/repositories/orgapacheflink-1178/
>>>>>>>
>>>>>>>          </url>
>>>>>>>        </repository>
>>>>>>>        <repository>
>>>>>>>          <id>archetype</id>
>>>>>>>          <url>
>>>>>>>
>>>>>>> https://repository.apache.org/content/repositories/orgapacheflink-1178/
>>>>>>>
>>>>>>>          </url>
>>>>>>>        </repository>
>>>>>>>      </repositories>
>>>>>>>    </profile>
>>>>>>> </profiles>
>>>>>>> </settings>
>>>>>>>
>>>>>>> And reference that in you maven commands via --settings
>>>>>>> path/to/settings.xml. This is useful for creating a quickstart
>>>>>>> based on
>>>>> the
>>>>>>> staged release and for building against the staged jars.
>>
>
>