git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Pinning dependencies for Apache Airflow


Hi Jarek,

I will +1 the discussion Dan is referring to and George's advice.

I just want to double check we are talking about pinning in
requirements.txt only.

This offers the ability to
pip install -r requirements.txt
pip install --no-deps airflow
For a guaranteed install which works.

Several different requirement files can be provided for specific use cases,
like a stable dev one for instance for people wanting to work on operators
and non-core functions.

However, I think we should proactively test in CI against unpinned
dependencies (though it might be a separate case in the matrix) , so that
we get advance warning if possible that things will break.
CI downtime is not a bad thing here, it actually caught a problem :)

We should unpin as possible in setup.py to only maintain minimum required
compatibility. The process of pinning in setup.py is extremely detrimental
when you have a large number of python libraries installed with different
pinned versions.

Best,
Arthur

On Thu, Oct 4, 2018 at 8:36 AM Dan Davydov <ddavydov@xxxxxxxxxxx.invalid>
wrote:

> Relevant discussion about this:
>
> https://github.com/apache/incubator-airflow/pull/1809#issuecomment-257502174
>
> On Thu, Oct 4, 2018 at 11:25 AM Jarek Potiuk <Jarek.Potiuk@xxxxxxxxxxx>
> wrote:
>
> > TL;DR; A change is coming in the way how dependencies/requirements are
> > specified for Apache Airflow - they will be fixed rather than flexible
> (==
> > rather than >=).
> >
> > This is follow up after Slack discussion we had with Ash and Kaxil -
> > summarising what we propose we'll do.
> >
> > *Problem:*
> > During last few weeks we experienced quite a few downtimes of TravisCI
> > builds (for all PRs/branches including master) as some of the transitive
> > dependencies were automatically upgraded. This because in a number of
> > dependencies we have  >= rather than == dependencies.
> >
> > Whenever there is a new release of such dependency, it might cause chain
> > reaction with upgrade of transitive dependencies which might get into
> > conflict.
> >
> > An example was Flask-AppBuilder vs flask-login transitive dependency with
> > click. They started to conflict once AppBuilder has released version
> > 1.12.0.
> >
> > *Diagnosis:*
> > Transitive dependencies with "flexible" versions (where >= is used
> instead
> > of ==) is a reason for "dependency hell". We will sooner or later hit
> other
> > cases where not fixed dependencies cause similar problems with other
> > transitive dependencies. We need to fix-pin them. This causes problems
> for
> > both - released versions (cause they stop to work!) and for development
> > (cause they break master builds in TravisCI and prevent people from
> > installing development environment from the scratch.
> >
> > *Solution:*
> >
> >    - Following the old-but-good post
> >    https://nvie.com/posts/pin-your-packages/ we are going to fix the
> > pinned
> >    dependencies to specific versions (so basically all dependencies are
> >    "fixed").
> >    - We will introduce mechanism to be able to upgrade dependencies with
> >    pip-tools (https://github.com/jazzband/pip-tools). We might also
> take a
> >    look at pipenv: https://pipenv.readthedocs.io/en/latest/
> >    - People who would like to upgrade some dependencies for their PRs
> will
> >    still be able to do it - but such upgrades will be in their PR thus
> they
> >    will go through TravisCI tests and they will also have to be specified
> > with
> >    pinned fixed versions (==). This should be part of review process to
> > make
> >    sure new/changed requirements are pinned.
> >    - In release process there will be a point where an upgrade will be
> >    attempted for all requirements (using pip-tools) so that we are not
> > stuck
> >    with older releases. This will be in controlled PR environment where
> > there
> >    will be time to fix all dependencies without impacting others and
> likely
> >    enough time to "vet" such changes (this can be done for alpha/beta
> > releases
> >    for example).
> >    - As a side effect dependencies specification will become far simpler
> >    and straightforward.
> >
> > Happy to hear community comments to the proposal. I am happy to take a
> lead
> > on that, open JIRA issue and implement if this is something community is
> > happy with.
> >
> > J.
> >
> > --
> >
> > *Jarek Potiuk, Principal Software Engineer*
> > Mobile: +48 660 796 129
> >
>