[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Use KubernetesExecutor to launch tasks into a Dask cluster in Kubernetes

I don't have a Dask cluster yet, but I'm interested in taking advantage of
it for ML tasks. My use case would be bursting a lot of ML jobs into a
Dask cluster all at once.
>From what I understand, Dask clusters utilize caching to help speed up jobs
so I don't know if it makes sense to launch a Dask cluster for every single
ML job. Conceivably, I could just have a single Dask worker running 24/7
and when its time to burst k8 could autoscale the Dask workers as more ML
jobs are launched into the Dask cluster?

On Fri, Apr 27, 2018 at 10:35 PM Daniel Imberman <daniel.imberman@xxxxxxxxx>

> Hi Kyle,
> So you have a static Dask cluster running your k8s cluster? Is there any
> reason you wouldn't just launch the Dask cluster for the job you're running
> and then tear it down? I feel like with k8s the elasticity is one of the
> main benefits.
> On Fri, Apr 27, 2018 at 12:32 PM Kyle Hamlin <> wrote:
> > Hi all,
> >
> > If I have a Kubernetes cluster running in DCOC and a Dask cluster running
> > in that same Kubernetes cluster is it possible/does it makes sense to use
> > the KubernetesExecutor to launch tasks into the Dask cluster (these are
> ML
> > jobs with sklearn)? I feel like there is a bit of inception going on here
> > in my mind and I just want to make sure a setup like this makes sense?
> > Thanks in advance for anyone's input!
> >

Kyle Hamlin

( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-airflow-development/msg03213.html on line 104
Call Stack
10.0015364696{main}( ).../msg03213.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-airflow-development/msg03213.html on line 104
Call Stack
10.0015364696{main}( ).../msg03213.html:0