[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Multiple hosts for a single connection

Hello -

I'd like to define multiple connections for the `webhdfs_default` conn_id,
as we have a primary and backup host for the hadoop namenode. This is for
use with the WebHDFSHook, which queries for connections and seemingly can
accept a list of them from the database.

        nn_connections = self.get_connections(self.webhdfs_conn_id)
        for nn in nn_connections:

I can manually add 2 entries for `webhdfs_default` in the Web UI, one for
each hostname, however with the CLI and adding the entries to the
airflow.cfg I get errors that a connection already exists.

It seems WebHDFSHook doesn't support reading multiple hosts from one entry.
Is this a bug? Or is there a different way to accomplish without manually
adding via the UI?

Ben Laird

( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-airflow-development/msg04151.html on line 91
Call Stack
10.0006363640{main}( ).../msg04151.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-airflow-development/msg04151.html on line 91
Call Stack
10.0006363640{main}( ).../msg04151.html:0