git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Pinning dependencies for Apache Airflow


> TL;DR; A change is coming in the way how dependencies/requirements are
> specified for Apache Airflow - they will be fixed rather than flexible (==
> rather than >=).

> This is follow up after Slack discussion we had with Ash and Kaxil -
> summarising what we propose we'll do.

Hey all.  It's great that we're moving this discussion back from Slack
to the mailing list.  But I've gotta point out that the wording needs
a small but critical fix up:

"A change *is* coming... they *will* be fixed"

needs to be

"We'd like to propose a change... We would like to make them fixed."

The first says that this decision has been made and the result of the
decision, which was made on Slack, is being reported back to the
mailing list.  The second is more accurate to the rest of the
discussion ('what we propose...').  And again, since it's axiomatic in
ASF that if it didn't happen on a list, it didn't happen[1], we gotta
make sure there's no confusion about where the community is on the
decision-making process.

Thanks,
Jakob

[1] https://community.apache.org/newbiefaq.html#NewbieFAQ-IsthereaCodeofConductforApacheprojects?
On Thu, Oct 4, 2018 at 9:56 AM Alex Guziel
<alex.guziel@xxxxxxxxxx.invalid> wrote:
>
> You should run `pip check` to ensure no conflicts. Pip does not do this on
> its own.
>
> On Thu, Oct 4, 2018 at 9:20 AM Jarek Potiuk <Jarek.Potiuk@xxxxxxxxxxx>
> wrote:
>
> > Great that this discussion already happened :). Lots of useful things in
> > it. And yes - it means pinning in requirement.txt - this is how pip-tools
> > work.
> >
> > J.
> >
> > Principal Software Engineer
> > Phone: +48660796129
> >
> > On Thu, 4 Oct 2018, 18:14 Arthur Wiedmer, <arthur.wiedmer@xxxxxxxxx>
> > wrote:
> >
> > > Hi Jarek,
> > >
> > > I will +1 the discussion Dan is referring to and George's advice.
> > >
> > > I just want to double check we are talking about pinning in
> > > requirements.txt only.
> > >
> > > This offers the ability to
> > > pip install -r requirements.txt
> > > pip install --no-deps airflow
> > > For a guaranteed install which works.
> > >
> > > Several different requirement files can be provided for specific use
> > cases,
> > > like a stable dev one for instance for people wanting to work on
> > operators
> > > and non-core functions.
> > >
> > > However, I think we should proactively test in CI against unpinned
> > > dependencies (though it might be a separate case in the matrix) , so that
> > > we get advance warning if possible that things will break.
> > > CI downtime is not a bad thing here, it actually caught a problem :)
> > >
> > > We should unpin as possible in setup.py to only maintain minimum required
> > > compatibility. The process of pinning in setup.py is extremely
> > detrimental
> > > when you have a large number of python libraries installed with different
> > > pinned versions.
> > >
> > > Best,
> > > Arthur
> > >
> > > On Thu, Oct 4, 2018 at 8:36 AM Dan Davydov <ddavydov@xxxxxxxxxxx.invalid
> > >
> > > wrote:
> > >
> > > > Relevant discussion about this:
> > > >
> > > >
> > >
> > https://github.com/apache/incubator-airflow/pull/1809#issuecomment-257502174
> > > >
> > > > On Thu, Oct 4, 2018 at 11:25 AM Jarek Potiuk <Jarek.Potiuk@xxxxxxxxxxx
> > >
> > > > wrote:
> > > >
> > > > > TL;DR; A change is coming in the way how dependencies/requirements
> > are
> > > > > specified for Apache Airflow - they will be fixed rather than
> > flexible
> > > > (==
> > > > > rather than >=).
> > > > >
> > > > > This is follow up after Slack discussion we had with Ash and Kaxil -
> > > > > summarising what we propose we'll do.
> > > > >
> > > > > *Problem:*
> > > > > During last few weeks we experienced quite a few downtimes of
> > TravisCI
> > > > > builds (for all PRs/branches including master) as some of the
> > > transitive
> > > > > dependencies were automatically upgraded. This because in a number of
> > > > > dependencies we have  >= rather than == dependencies.
> > > > >
> > > > > Whenever there is a new release of such dependency, it might cause
> > > chain
> > > > > reaction with upgrade of transitive dependencies which might get into
> > > > > conflict.
> > > > >
> > > > > An example was Flask-AppBuilder vs flask-login transitive dependency
> > > with
> > > > > click. They started to conflict once AppBuilder has released version
> > > > > 1.12.0.
> > > > >
> > > > > *Diagnosis:*
> > > > > Transitive dependencies with "flexible" versions (where >= is used
> > > > instead
> > > > > of ==) is a reason for "dependency hell". We will sooner or later hit
> > > > other
> > > > > cases where not fixed dependencies cause similar problems with other
> > > > > transitive dependencies. We need to fix-pin them. This causes
> > problems
> > > > for
> > > > > both - released versions (cause they stop to work!) and for
> > development
> > > > > (cause they break master builds in TravisCI and prevent people from
> > > > > installing development environment from the scratch.
> > > > >
> > > > > *Solution:*
> > > > >
> > > > >    - Following the old-but-good post
> > > > >    https://nvie.com/posts/pin-your-packages/ we are going to fix the
> > > > > pinned
> > > > >    dependencies to specific versions (so basically all dependencies
> > are
> > > > >    "fixed").
> > > > >    - We will introduce mechanism to be able to upgrade dependencies
> > > with
> > > > >    pip-tools (https://github.com/jazzband/pip-tools). We might also
> > > > take a
> > > > >    look at pipenv: https://pipenv.readthedocs.io/en/latest/
> > > > >    - People who would like to upgrade some dependencies for their PRs
> > > > will
> > > > >    still be able to do it - but such upgrades will be in their PR
> > thus
> > > > they
> > > > >    will go through TravisCI tests and they will also have to be
> > > specified
> > > > > with
> > > > >    pinned fixed versions (==). This should be part of review process
> > to
> > > > > make
> > > > >    sure new/changed requirements are pinned.
> > > > >    - In release process there will be a point where an upgrade will
> > be
> > > > >    attempted for all requirements (using pip-tools) so that we are
> > not
> > > > > stuck
> > > > >    with older releases. This will be in controlled PR environment
> > where
> > > > > there
> > > > >    will be time to fix all dependencies without impacting others and
> > > > likely
> > > > >    enough time to "vet" such changes (this can be done for alpha/beta
> > > > > releases
> > > > >    for example).
> > > > >    - As a side effect dependencies specification will become far
> > > simpler
> > > > >    and straightforward.
> > > > >
> > > > > Happy to hear community comments to the proposal. I am happy to take
> > a
> > > > lead
> > > > > on that, open JIRA issue and implement if this is something community
> > > is
> > > > > happy with.
> > > > >
> > > > > J.
> > > > >
> > > > > --
> > > > >
> > > > > *Jarek Potiuk, Principal Software Engineer*
> > > > > Mobile: +48 660 796 129
> > > > >
> > > >
> > >
> >