git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Benchmarking of Airflow Scheduler with Celery Executor


Thanks Ry,
Just wondering if there is any approximate number on concurrent tasks a scheduler can run on say 16 GB RAM and 8 core machine.
If its already been done that would be useful.
We did some benchmarking with local executor and observed that each TaskInstance was taking ~100MB of memory so we could only run ~130 concurrent tasks on 16 GB RAM and 8 core machine.

-Raman Gupta  

 

On 2018/04/12 16:32:37, Ry Walker <ry@xxxxxxxxxxxxx> wrote: 
> Hi Raman -
> 
> First, we’d be happy to help you test this out with Airflow. Or you could
> do it yourself by using http://open.astronomer.io/airflow/ (w/ Docker
> Engine + Docker Compose) to quickly spin up a test environment. Everything
> is hooked to Prometheus/Grafana to monitor how the system reacts to your
> workload.
> 
> -Ry
> CEO, Astronomer
> 
> On April 12, 2018 at 12:23:46 PM, ramandumcs@xxxxxxxxx (ramandumcs@xxxxxxxxx)
> wrote:
> 
> Hi All,
> We have requirement to run 10k(s) of concurrent tasks. We are exploring
> Airflow's Celery Executor for same. Horizontally Scaling of worker nodes
> seem possible but it can only have one active scheduler.
> So will Airflow scheduler be able to handle these many concurrent tasks.
> Is there any benchmarking number around airflow scheduler's scalability.
> Thanks,
> Raman
> 



( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-airflow-development/msg03132.html on line 102
Call Stack
#TimeMemoryFunctionLocation
10.0006363672{main}( ).../msg03132.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-airflow-development/msg03132.html on line 102
Call Stack
#TimeMemoryFunctionLocation
10.0006363672{main}( ).../msg03132.html:0