git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

How to consolidate log files?


Hi Mailing List,

Is there a way to consolidate airflow task logs by stdout and stderr? Currently, the structure is something like /logs/taskname/task_run_date/log1.txt which is the log for a particular taskname at a particular run date.  What I would like is two large log files for all tasks, something like /logs/errors.txt and /logs/outputs.txt  Which would contain all the stderr and stdout messages for all runs of all tasks regardless of run date. I essentially want two very large log files.  For example, if I have task A and task B, instead of having two directories and then subdirectories for A and B, I would just like two files one with errors from A and B and one for outputs from A and B. Does airflow provide this information?

Thanks!