Re: Use KubernetesExecutor to launch tasks into a Dask cluster in Kubernetes
Also one of the main benefits of the Kubernetes Executor is having a Docker
image that contains all the dependencies that you need for your job.
Personally I would switch to Kubernetes when it leaves the experimental
2018-04-28 16:27 GMT+02:00 Kyle Hamlin <hamlin.kn@xxxxxxxxx>:
> I don't have a Dask cluster yet, but I'm interested in taking advantage of
> it for ML tasks. My use case would be bursting a lot of ML jobs into a
> Dask cluster all at once.
> From what I understand, Dask clusters utilize caching to help speed up jobs
> so I don't know if it makes sense to launch a Dask cluster for every single
> ML job. Conceivably, I could just have a single Dask worker running 24/7
> and when its time to burst k8 could autoscale the Dask workers as more ML
> jobs are launched into the Dask cluster?
> On Fri, Apr 27, 2018 at 10:35 PM Daniel Imberman <
> > Hi Kyle,
> > So you have a static Dask cluster running your k8s cluster? Is there any
> > reason you wouldn't just launch the Dask cluster for the job you're
> > and then tear it down? I feel like with k8s the elasticity is one of the
> > main benefits.
> > On Fri, Apr 27, 2018 at 12:32 PM Kyle Hamlin <hamlin.kn@xxxxxxxxx>
> > > Hi all,
> > >
> > > If I have a Kubernetes cluster running in DCOC and a Dask cluster
> > > in that same Kubernetes cluster is it possible/does it makes sense to
> > > the KubernetesExecutor to launch tasks into the Dask cluster (these are
> > ML
> > > jobs with sklearn)? I feel like there is a bit of inception going on
> > > in my mind and I just want to make sure a setup like this makes sense?
> > > Thanks in advance for anyone's input!
> > >
> Kyle Hamlin