Re: [DISCUSS] Re-think CI strategy?
A straightforward way would be to run non-critical CI jobs as nightlies.
Nightly package builds work pretty well, see the following link
the notification logic requires improvement though.
We should also run integrations tests regularly (dask, hdfs, spark).
If We use a non apache queue repository for crossbow, We could even
submit jobs to other CI services (e.g. CircleCI) to increase the build
On Aug 6 2018, at 1:23 pm, Antoine Pitrou <antoine@xxxxxxxxxx> wrote:
> Our CI jobs are taking longer and longer. The main reason seem not to
> be that our test suites become more thorough (running tests actually
> seems to account for a very minor fraction of CI times) but the combined
> fact that 1) fetching dependencies and building is slow 2) we have many
> configurations tested on our CI jobs.
> The slowest job on our Travis-CI configuration routinely takes more 40
> minutes (this is for a single job, where everything is essentially
> serialized). As for AppVeyor, the different jobs in a build there are
> mostly serialized, which balloons the total build time.
> If we ever move parquet-cpp into the Arrow repo, this will probably
> become worse again (though it might be better for parquet-cpp itself,
> which doesn't seem to have received a lot of care on the CI side: for
> instance, parquet-cpp AppVeyor builds are abysmally slow).
> I think we will have to cut down on the number of things we exercise on
> the per-build CI builds. This can be done along two axes:
> 1) remove some build steps that we deem non-critical or even unimportant
> 2) remove some build configurations entirely (for instance I don't
> understand why we need a "static CRT" build on Windows, or worse, why we
> need to have NMake-based builds at all)
> Any thoughts?