git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Single Airflow Instance Vs Multiple Airflow Instance


So I would guess you have multiple team and could you give me an idea of
how many DAGs you actually have and how many runs per day ?

On Wed, Jun 6, 2018 at 11:30 AM, Arturo Michel <Arturo.Michel@xxxxxxxxxxxxxx
> wrote:

> We incorportated multi-tentacy in how we deploy dags.
>
> With version 0.9 you can have the following structure:
> DAG_HOME/
>  Project1/
>   - Dag_1
>   - Lib_for_dag_1
>  Project2/
>   - Dag_1
>   - lib_for_dag_1
>
> So that the team owning Project1 can independently and without inferring
> with others deploy (Project2) their own dags. If you incorporate this
> structure in your deployment process you could do this.
>
> It works fine.
>
> -----Original Message-----
> From: gauthiermartin86@ <gmail.com gauthiermartin86@xxxxxxxxx>
> Sent: Wednesday, June 6, 2018 4:24 PM
> To: dev@xxxxxxxxxxxxxxxxxx
> Subject: Single Airflow Instance Vs Multiple Airflow Instance
>
> Hi Everyone,
>
> We have been experimenting with airflow for about 6 months now.
> We are planning to have multiple departments to use it. Since we don't
> have any internal experience with Airflow we are wondering if single
> instance per department is more suited than single instance with
> multi-tenancy? We have been aware about the upcoming release of airflow
> 1.10 and changes that will be made to the RBAC which will be more suited
> for multi-tenancy.
>
> Any advice on this ? Any tips could be helpful to us.
>
> This e-mail message and any attachments are confidential and are for the
> exclusive use of the addressee only.  If you are not the intended
> recipient, you should not use the content, place any reliance on it or
> disclose it to anyone else.  Please notify the sender immediately by
> replying to it and then ensure that it is deleted from your system
> (including any attachments).
>



-- 
Martin Gauthier

C: 438-499-1749


( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-airflow-development/msg03586.html on line 123
Call Stack
#TimeMemoryFunctionLocation
10.0006358472{main}( ).../msg03586.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-airflow-development/msg03586.html on line 123
Call Stack
#TimeMemoryFunctionLocation
10.0006358472{main}( ).../msg03586.html:0