git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Assigning multiple pools to a task


A task is assigned to a pool by the task specifying the name of the pool.
The docs suggest that the pool argument is a string, not a list of strings.
https://airflow.apache.org/code.html#baseoperator

And looking at the code it does seem like this relationship of one task
assigned to zero or one pools is strict.

It's an interesting idea of trying to associated a task with multiple
pools.  I think things would get complicated fast to avoid deadlock
situations.  Do you know of any other work flow systems that support
one-to-many relationship?

Is it possible to bundle the resources into a single resource?  I'm
guessing this is not an option, and that tasks use different combinations
of these resources.

cheers,
Dennis


On Fri, May 11, 2018 at 1:04 PM Neelesh <neeleshs@xxxxxxxxx> wrote:

> Hello,
>
> Is it possible to assign multiple pools to a task?
>
> I'm wrapping a bash operator around some heavy-weight custom data
> processing script that cannot be modified. This script cannot be modified,
> and it uses multiple resources. Im trying to use pools as proxies for these
> resources.
>
> Thanks
>