How to consolidate log files?
Hi Mailing List,
Is there a way to consolidate airflow task logs by stdout and stderr? Currently, the structure is something like /logs/taskname/task_run_date/log1.txt which is the log for a particular taskname at a particular run date. What I would like is two large log files for all tasks, something like /logs/errors.txt and /logs/outputs.txt Which would contain all the stderr and stdout messages for all runs of all tasks regardless of run date. I essentially want two very large log files. For example, if I have task A and task B, instead of having two directories and then subdirectories for A and B, I would just like two files one with errors from A and B and one for outputs from A and B. Does airflow provide this information?