git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Disable Processing of DAG file


Hi folks,
This config line
<https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L414>
controls how often the scheduler scan the DAG folder and tries to discover/
forget DAGs.

For doing dag file processing part, scheduler does parse the DAG file
everytime before it schedules tasks through DagFileProcessor.

Cheers,
Kevin Y

On Mon, May 28, 2018 at 10:14 PM, Ananth Durai <vananth22@xxxxxxxxx> wrote:

> It is an interesting question. On a slightly related note, Correct me if
> I'm wrong, AFAIK we require restarting airflow scheduler in order pick any
> new DAG file changes by the scheduler. In that case, should the scheduler
> do the DAGFileProcessing every time before scheduling the tasks?
>
> Regards,
> Ananth.P,
>
>
>
>
>
>
> On 28 May 2018 at 21:46, ramandumcs@xxxxxxxxx <ramandumcs@xxxxxxxxx>
> wrote:
>
> > Hi All,
> >
> > We have a use case where there would be 100(s) of DAG files with schedule
> > set to "@once". Currently it seems that scheduler processes each and
> every
> > file and creates a Dag Object.
> > Is there a way or config to tell scheduler to stop processing certain
> > files.
> >
> > Thanks,
> > Raman Gupta
> >
>