git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [DISCUSS] Re-think CI strategy?


Hi,

A straightforward way would be to run non-critical CI jobs as nightlies.
Nightly package builds work pretty well, see the following link
https://github.com/kszucs/crossbow/branches/all?query=nightly
the notification logic requires improvement though.
We should also run integrations tests regularly (dask, hdfs, spark).
If We use a non apache queue repository for crossbow, We could even
submit jobs to other CI services (e.g. CircleCI) to increase the build
parallelism.

Krisztian
On Aug 6 2018, at 1:23 pm, Antoine Pitrou <antoine@xxxxxxxxxx> wrote:
>
>
> Hello,
> Our CI jobs are taking longer and longer. The main reason seem not to
> be that our test suites become more thorough (running tests actually
> seems to account for a very minor fraction of CI times) but the combined
> fact that 1) fetching dependencies and building is slow 2) we have many
> configurations tested on our CI jobs.
>
> The slowest job on our Travis-CI configuration routinely takes more 40
> minutes (this is for a single job, where everything is essentially
> serialized). As for AppVeyor, the different jobs in a build there are
> mostly serialized, which balloons the total build time.
>
> If we ever move parquet-cpp into the Arrow repo, this will probably
> become worse again (though it might be better for parquet-cpp itself,
> which doesn't seem to have received a lot of care on the CI side: for
> instance, parquet-cpp AppVeyor builds are abysmally slow).
>
> I think we will have to cut down on the number of things we exercise on
> the per-build CI builds. This can be done along two axes:
> 1) remove some build steps that we deem non-critical or even unimportant
> 2) remove some build configurations entirely (for instance I don't
> understand why we need a "static CRT" build on Windows, or worse, why we
> need to have NMake-based builds at all)
>
> Any thoughts?
> Regards
> Antoine.