Re: Proposed changes to CircleCI testing workflow
Just want to say +1, and thanks for taking the time to do this. This all sounds great.
(And completely supersedes what I hoped to achieve with CASSANDRA-14648)
> On 26 Oct 2018, at 15:49, Stefan Podkowinski <spod@xxxxxxxxxx> wrote:
> I'd like to give you a quick update on the work that has been done
> lately on running tests using CircleCI. Please let me know if you have
> any objections or don't think this is going into the right direction, or
> have any other feedback!
> We've been using CircleCI for a while now and results are used on
> constant basis for new patches. Not only by committers, but also by
> casual contributors to run unit tests. Looks like people find the
> service valuable and we should keep using it. Therefor I'd like to make
> some improvements that will make it easier to add new tests and to
> continue making CircleCI an option for all contributors, both on paid
> and free plans.
> The general idea of the changes implemented in #14806, is to consolidate
> the existing config to make it more modular and have smaller jobs that
> can be scheduled ad-hoc by the developer, instead of running a few big
> jobs on every commit. Reorganizing and breaking up the existing config
> was done using the new 2.1 config features. Starting jobs on request,
> instead of automatically, is done using the manual approval feature,
> i.e. you now have to click on that job in the workflow page in order to
> start it. I'd like to see us having smaller, more specialized groups of
> tests that we can run more selectively during development, while still
> being able to run bigger tests before committing, or firing up all of
> them during testing and releasing. Other example of smaller jobs would
> be testing coverage (#14788) or cqlsh tests (#14298). But also
> individual jobs for different ant targets, like burn, stress or benchmarks.
> We'd now also be able to run tests using different docker images and
> different JDKs. I've already updated the used image to also include Java
> 11 and added unit and dtest jobs to the config for that. It's now really
> easy to run tests on Java 11, although these won't pass yet. It seems to
> be important to me to have this kind of flexibility, given the
> increasingly diverse ecosystem of Java distributions. We can also add
> jobs for packaging and doing smoke tests by installing and starting
> packages on different docker images (Redhat, Debian, Ubuntu,..) at a
> later point.
> As for the paid vs free plans issue, I'd also like us to discuss how we
> can make tests faster and less resource intensive in general. As a
> desired consequence, we'd be able to move away from multi-node dtests,
> to something that can be run using the free plan. I'm looking forward to
> see if #14821 can get us into that direction. Ideally we can add these
> tests into a job that can be completed on the free plan and encourage
> contributors to add new tests there, instead of having to write a dtest,
> which they won't be able to run on CircleCI without a paid plan.
> Whats changing for you as a CircleCI user?
> * All tests, except unit tests, will need to be started manually and
> will not run on every commit (this can be further discussed and changed
> anytime if needed)
> * Updating the config.yml file now requires using the CircleCI cli tool
> and should not be done directly (see #14806 for technical details)
> * High resource settings can be enabled using a script/patch, either run
> manually or as commit hook (again see ticket for details)
> * Both free and paid plan users now have more tests to run
> As already mentioned, please let me know if you have any thoughts on
> this, or if you think this is going into the wrong direction.
> To unsubscribe, e-mail: dev-unsubscribe@xxxxxxxxxxxxxxxxxxxx
> For additional commands, e-mail: dev-help@xxxxxxxxxxxxxxxxxxxx
To unsubscribe, e-mail: dev-unsubscribe@xxxxxxxxxxxxxxxxxxxx
For additional commands, e-mail: dev-help@xxxxxxxxxxxxxxxxxxxx