Re: [DISCUSS] Re-think CI strategy?
On Mon, Aug 6, 2018 at 7:52 AM, Antoine Pitrou <antoine@xxxxxxxxxx> wrote:
> Le 06/08/2018 à 13:42, Wes McKinney a écrit :
>> hi Antoine,
>> I completely agree. Part of why I've been so consistently pressing for
>> nightly build tooling is to be able to shift more exhaustive testing
>> out of per-commit runs into a daily build or an on-demand build to be
>> invoked by the user either manually or by means of a bot. If you
>> search in JIRA for the term "nightly" you can see a lot of issue that
>> I have created for this already.
> My worry with nightly jobs is that they doesn't clearly pinpoint the
> specific changeset where a regression occurred; also, since it happens
> after merging, there is less incentive to clear any mess introduced by a
> PR; moreover, it makes trying out a fix more difficult, since you don't
> get direct feedback on a commit or PR.
> So in exchange for lowering per-commit CI times, nightly jobs require
> extra human care to track regressions and make sure they get fixed.
The way that projects with much more complex testing deal with this is
to have a pre-commit test run. The Apache Impala "verify merge" commit
take several hours to run, for example. I don't think we are on our
way to that point yet.
What I'm suggesting is that we be able to write
@arrow-test-bot please run exhaustive
There are a lot of patches that clearly don't require exhaustive
testing. In any case, we are likely to accumulate more testing than we
can run on every commit / patch, but as long as we have a convenient
mechanism to run the tests, then it is OK.
>> it would be useful to be
>> able to validate if desired (and in an automated way) that the NMake
>> build works properly
> I guess the main question is why we're testing for NMake at all. CMake
> supports a range of different build tools, we can't exercise all of
> them. So I'd say on each platform we should exercise at most two build
> - Ninja, because it's cross-platform and the fastest (which makes it
> desirable for developers *and* for CI)
> - the standard platform-specific build tool, i.e. GNU make on Unix and
> Visual Studio (or "msbuild") on Windows
Well, I would be OK with dropping NMake in favor of supporting Ninja.
>> I think we can also improve CI build times by caching certain
>> toolchain artifacts that are taking a long time to build (Thrift, I'm
>> looking at you).
> The 40+ minutes Travis-CI job already uses the toolchain packages AFAIK.
> Don't they include thrift?
The most time consuming parts of the job are AFAIK:
* The asv benchmarks
* Code coverage uploads
* Using valgrind
* Building multiple versions of Python
These can be addressed respectively by:
* Nightly/opt-in testing asv (via bot)
* Nightly/opt-in coverage
* Nightly valgrind
* No immediate solution. I would like to drop Python 2 in January 2019
> (another thought: when do we want to drop Python 2 compatibility?)