git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Subdags getting created partially


Hi All,

We are trying to create a set of subdags, given a parent dag, which are to
be created and executed in batches.

The following piece of code creates the subdags:
def create_subdag(instr_test, check_batch_size, batch_start, batch_end):
    instr_subdag_func = SUB_DAG_PRE + "_" + str(instr_test)
    instr_subdag_id = SUB_DAG_PRE + "-" + str(instr_test)

    instr_subdag_func = SubDagOperator(
        task_id=instr_subdag_id,
        subdag=instr_subdag_csv(MAIN_DAG_ID, instr_subdag_id, args,
instr_test),
        default_args=args,
        dag=dag)

    instr_subdag_func.set_upstream(batch_start)
    batch_end.set_upstream(instr_subdag_func)
    check_batch_size = check_batch_size + 1

    return check_batch_size

However, it seems that given a total no. of subdags to be created, not all
are gettng created. In our case, out of 54, only 38 are getting created.
Can someone please advise as to what might be going wrong ?

Also, do subdags also get listed out when we run the command : airflow
list_dags ? Just wanted to confirm if that is expected.

Sajan


( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-airflow-development/msg03589.html on line 99
Call Stack
#TimeMemoryFunctionLocation
10.0007358472{main}( ).../msg03589.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-airflow-development/msg03589.html on line 99
Call Stack
#TimeMemoryFunctionLocation
10.0007358472{main}( ).../msg03589.html:0