git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Airflow Docker Container


As Scott pointed out, we at Astronomer.io have taken the lightweight image
philosophy to heart with our docker images. Our base image inherits from
Alpine
(https://github.com/astronomerio/astronomer) and our Airflow image (
https://github.com/astronomerio/astronomer/blob/master/docker/platform/airflow/Dockerfile)
layers on that. We then have another layer (
https://github.com/astronomerio/astronomer/blob/master/docker/platform/airflow/onbuild/Dockerfile)
which can be used to install system level packages and python dependencies
from packages.txt and requirements.txt.

If, for some reason, these images don't check all your boxes, let us know
and we'll see what we can do to make them better for you.

-Andy Cooper

On Mon, May 14, 2018 at 4:44 PM Joe Napolitano <joe.napolitano@xxxxxxxxxx>
wrote:

> You may consider this base image we put together at Blue Apron. My fork
> fixes a build issue by pinning to pip < 10.
>
> https://github.com/joenap/airflow-base
>
> Joe Nap
>
> On Mon, May 14, 2018 at 4:37 PM, Daniel Imberman <
> daniel.imberman@xxxxxxxxx>
> wrote:
>
> > @Fokko
> >
> > I definitely agree with that. I think that having a "super lightweight"
> > image for just running a basic airflow instance makes sense. We could
> even
> > name the image something like  airflow-k8s so people know it's ONLY meant
> > to work in a k8s cluster. I'm trying to figure out what methods besides
> > helm we should be considering (Helm doesn't really have full saturation
> in
> > the k8s world so wanna see if there are other deployment tools we should
> > consider).
> >
> > @Scott Dang quite a bit is definitely an understatement :). Would anyone
> on
> > your team have some cycles to work with @jzucker or @sedwards on the
> > helm/deployment stuff?
> >
> > On Mon, May 14, 2018 at 1:18 PM Driesprong, Fokko <fokko@xxxxxxxxxxxxxx>
> > wrote:
> >
> > > Hi Daniel,
> > >
> > > My dear colleague from GoDataDriven, Bas Harenslak, started on building
> > an
> > > official Docker container on the Dockerhub. I've put him in the CC. In
> > the
> > > end I strongly believe the image should end up in the official Docker
> > > repository: https://github.com/docker-library/official-images
> > >
> > > Right now, the excellent images provided by Puckel are widely used for
> > > running Airflow in Docker. For the Kubernetes build we need to pull in
> > some
> > > additional dependencies. Maybe a good idea to do this separately from
> the
> > > one from Puckel, to keep his images lightweight. Any thoughts?
> > >
> > > Kind regards,
> > > Fokko Driesprong
> > >
> > >
> > > 2018-05-14 22:09 GMT+02:00 Anirudh Ramanathan <
> > > ramanathana@xxxxxxxxxx.invalid>:
> > >
> > >> @Erik Erlandson <eje@xxxxxxxxxx> has had conversations about
> publishing
> > >
> > >
> > >> docker images with the ASF Legal team.
> > >> Adding him to the thread.
> > >>
> > >> On Mon, May 14, 2018 at 1:07 PM Daniel Imberman <
> > >> daniel.imberman@xxxxxxxxx>
> > >> wrote:
> > >>
> > >> > Hi everyone,
> > >> >
> > >> > I've started looking into creating an official airflow docker
> > container
> > >> > s.t. users of the KubernetesExecutor could auto-pull from helm
> > >> > charts/deployment yamls/etc. I was wondering what everyone thinks
> the
> > >> best
> > >> > way to do this would be? Is there an official apache docker repo? Is
> > >> there
> > >> > a preferred linux distro?
> > >> >
> > >> > cc: @anirudh since this was something you had to deal with for
> > >> > spark-on-k8s.
> > >> >
> > >>
> > >>
> > >> --
> > >> Anirudh Ramanathan
> > >>
> > >
> >
>