git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

KubernetesPodOperator: Invalid arguments were passed to BaseOperator


I tested master branch by putting the following in my requirements.txt:

git+https://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]

and did a pip install -r requirements.txt

When I started the airflow webserver, I saw deprecation warnings.  I
put some additional debugging in models.py to through an exception so that I could see the
full stacktrace:

[2018-05-29 14:00:34,419] {models.py:307} ERROR - Failed to import: /Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/example_dags/example_kubernetes_operator.py
Traceback (most recent call last):
  File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/models.py", line 304, in process_file
    m = imp.load_source(mod_name, filepath)
  File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/example_dags/example_kubernetes_operator.py", line 53, in <module>
    dag=dag)
  File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/utils/decorators.py", line 98, in wrapper
    result = func(*args, **kwargs)
  File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/models.py", line 2308, in __init__
    raise Exception("Invalid use of args or kwargs")
Exception: Invalid use of args or kwargs


If looks like example_kubernetes_operator.py, this code is the source of the exception:

k = KubernetesPodOperator(
    namespace='default',
    image="ubuntu:16.04",
    cmds=["bash", "-cx"],
    arguments=["echo", "10"],
    labels={"foo": "bar"},
    name="airflow-test-pod",
    in_cluster=False,
    task_id="task",
    get_logs=True,
    dag=dag)


Without my extra debugging, the deprecation warning looks like this:

[2018-05-29 14:06:27,567] {example_kubernetes_operator.py:30} WARNING - Could not import KubernetesPodOperator
/Users/c-craigr/airflow2/lib/python2.7/site-packages/airflow/models.py:2315: PendingDeprecationWarning: Invalid arguments were passed to BaseOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were:
*args: ()
**kwargs: {'name': 'airflow-test-pod', 'image': 'ubuntu:16.04', 'labels': {'foo': 'bar'}, 'namespace': 'default', 'cmds': ['bash', '-cx'], 'arguments': ['echo', '10'], 'in_cluster': False, 'get_logs': True}
  category=PendingDeprecationWarning



What is the correct fix for this?  It looks like a lot of operators pass in arguments which are not
processed by BaseOperator, and thus trip over this deprecation warning.

--
Craig