git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Airflow 1.10 Migration Duration


Hi folks,

Here at Clover we're excitedly migrating to Airflow 1.10 (thanks for
everyone's hard work on that!). We're finding that it's taking about 2
hours to apply all the migrations to go from Airflow 1.8 to 1.10, largely
driven by the 10 million rows in our task_instance table. That got us
wondering what kind of maintenance people do on their Airflow metadata
databases. Do folks mostly put up with long migrations and generally longer
queries, or are y'all doing periodic cleanups of your metadata DB to keep
it fairly light?

Thanks,
Matt Davis