git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Airflow Docker Container


You may consider this base image we put together at Blue Apron. My fork
fixes a build issue by pinning to pip < 10.

https://github.com/joenap/airflow-base

Joe Nap

On Mon, May 14, 2018 at 4:37 PM, Daniel Imberman <daniel.imberman@xxxxxxxxx>
wrote:

> @Fokko
>
> I definitely agree with that. I think that having a "super lightweight"
> image for just running a basic airflow instance makes sense. We could even
> name the image something like  airflow-k8s so people know it's ONLY meant
> to work in a k8s cluster. I'm trying to figure out what methods besides
> helm we should be considering (Helm doesn't really have full saturation in
> the k8s world so wanna see if there are other deployment tools we should
> consider).
>
> @Scott Dang quite a bit is definitely an understatement :). Would anyone on
> your team have some cycles to work with @jzucker or @sedwards on the
> helm/deployment stuff?
>
> On Mon, May 14, 2018 at 1:18 PM Driesprong, Fokko <fokko@xxxxxxxxxxxxxx>
> wrote:
>
> > Hi Daniel,
> >
> > My dear colleague from GoDataDriven, Bas Harenslak, started on building
> an
> > official Docker container on the Dockerhub. I've put him in the CC. In
> the
> > end I strongly believe the image should end up in the official Docker
> > repository: https://github.com/docker-library/official-images
> >
> > Right now, the excellent images provided by Puckel are widely used for
> > running Airflow in Docker. For the Kubernetes build we need to pull in
> some
> > additional dependencies. Maybe a good idea to do this separately from the
> > one from Puckel, to keep his images lightweight. Any thoughts?
> >
> > Kind regards,
> > Fokko Driesprong
> >
> >
> > 2018-05-14 22:09 GMT+02:00 Anirudh Ramanathan <
> > ramanathana@xxxxxxxxxx.invalid>:
> >
> >> @Erik Erlandson <eje@xxxxxxxxxx> has had conversations about publishing
> >
> >
> >> docker images with the ASF Legal team.
> >> Adding him to the thread.
> >>
> >> On Mon, May 14, 2018 at 1:07 PM Daniel Imberman <
> >> daniel.imberman@xxxxxxxxx>
> >> wrote:
> >>
> >> > Hi everyone,
> >> >
> >> > I've started looking into creating an official airflow docker
> container
> >> > s.t. users of the KubernetesExecutor could auto-pull from helm
> >> > charts/deployment yamls/etc. I was wondering what everyone thinks the
> >> best
> >> > way to do this would be? Is there an official apache docker repo? Is
> >> there
> >> > a preferred linux distro?
> >> >
> >> > cc: @anirudh since this was something you had to deal with for
> >> > spark-on-k8s.
> >> >
> >>
> >>
> >> --
> >> Anirudh Ramanathan
> >>
> >
>