git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Airflow with Celery


What I find is that when celery rejects we hit this.  For us we don't do
work on the hosts so solve by over provisioning tasks in celery

On Tue, May 15, 2018, 6:30 AM Andy Cooper <andy.cooper@xxxxxxxxxxxxx> wrote:

> I have had very similar issues when there was a problem with the connection
> string pointing to the message broker. Triple check those connection
> strings and attempt to connect outside of airflow.
>
> On Tue, May 15, 2018 at 9:27 AM Goutham Pratapa <pratapagoutham@xxxxxxxxx>
> wrote:
>
> > Hi all,
> >
> > I have been using airflow with Celery executor in the background
> >
> > https://hastebin.com/sipecovomi.ini --> airflow.cfg
> >
> > https://hastebin.com/urutokuvoq.py   --> The dag I have been using
> >
> >
> >
> > This shows that the dag is always in running state.
> >
> >
> >
> >
> > Airflow flower shows nothing in the tasks or in the broker.
> >
> >
> > Did I miss anything can anyone help me in this regard.
> >
> >
> > --
> > Cheers !!!
> > Goutham Pratapa
> >
>