git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [VOTE] Airflow 1.10.0rc4


-1 (binding)

Sorry Bolke for not checking this earlier. In rc3 we've replaced some of
the reserved keywords. But still I'm unable to run a simple dag in the 1.10
rc4 release under Python 3.7:

MacBook-Pro-van-Fokko:sdh-api-pobt fokkodriesprong$ docker run -e
SLUGIFY_USES_TEXT_UNIDECODE=yes -t -i python:3.7 /bin/bash -c "pip install
https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc4/apache-airflow-1.10.0rc4+incubating-bin.tar.gz
&& airflow initdb && airflow run example_bash_operator runme_0 2017-07-01"
Collecting
https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc4/apache-airflow-1.10.0rc4+incubating-bin.tar.gz
  Downloading
https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc4/apache-airflow-1.10.0rc4+incubating-bin.tar.gz
(4.4MB)
    100% |████████████████████████████████| 4.4MB 2.5MB/s
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-91ci7xlu/setup.py", line 124
        async = [
              ^
    SyntaxError: invalid syntax

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in
/tmp/pip-req-build-91ci7xlu/
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

I think we should cherry-pick these three commits on to 1.10 branch:

*- Remove the async from setup.py*
https://github.com/apache/incubator-airflow/commit/e38a4e5d3064980abd10b8afa6918ab9f10dd8a2

*- Upgrade lxml to >4.0 to let it compile with Python 3.7*
https://github.com/apache/incubator-airflow/commit/5290688ee0576ad167d9622c96cdeb08e9965a20
lxml is needed for Python 3.7:
https://github.com/apache/incubator-airflow/pull/3583

*- Bump tenacy from 4.8.0 to 4.12.0 *
https://github.com/apache/incubator-airflow/pull/3723/commits/271ea663df72c16aa105017ed5cc87a639846777
The 4.8 version of Tenacy contains reserved keywords
https://github.com/apache/incubator-airflow/pull/3723

After this I'm able to run an example dag using Python3.7: docker run -e
SLUGIFY_USES_TEXT_UNIDECODE=yes -t -i python:3.7 /bin/bash -c "pip install
git+https://github.com/Fokko/incubator-airflow.git@v1-10-stable && airflow
initdb && airflow run example_bash_operator runme_0 2017-07-01"

[2018-08-08 21:25:57,944] {__init__.py:51} INFO - Using executor
SequentialExecutor
[2018-08-08 21:25:58,069] {models.py:258} INFO - Filling up the DagBag from
/root/airflow/dags
[2018-08-08 21:25:58,112] {example_kubernetes_operator.py:54} WARNING -
Could not import KubernetesPodOperator: No module named 'kubernetes'
[2018-08-08 21:25:58,112] {example_kubernetes_operator.py:55} WARNING -
Install kubernetes dependencies with:     pip install airflow['kubernetes']
[2018-08-08 21:25:58,155] {cli.py:492} INFO - Running <TaskInstance:
example_bash_operator.runme_0 2017-07-01T00:00:00+00:00 [None]> on host
31ec1d1554b7
[2018-08-08 21:25:58,739] {__init__.py:51} INFO - Using executor
SequentialExecutor
[2018-08-08 21:25:58,915] {models.py:258} INFO - Filling up the DagBag from
/root/airflow/dags/example_dags/example_bash_operator.py
[2018-08-08 21:25:58,987] {example_kubernetes_operator.py:54} WARNING -
Could not import KubernetesPodOperator: No module named 'kubernetes'
[2018-08-08 21:25:58,987] {example_kubernetes_operator.py:55} WARNING -
Install kubernetes dependencies with:     pip install airflow['kubernetes']
[2018-08-08 21:25:59,060] {cli.py:492} INFO - Running <TaskInstance:
example_bash_operator.runme_0 2017-07-01T00:00:00+00:00 [None]> on host
31ec1d1554b7

https://github.com/Fokko/incubator-airflow/commits/v1-10-stable

Still no hard guarantees that 3.7 will be fully supported, but at least it
runs :-)

Cheers, Fokko

2018-08-08 20:21 GMT+02:00 Bolke de Bruin <bdbruin@xxxxxxxxx>:

> Hey all,
>
> I have cut Airflow 1.10.0 RC4. This email is calling a vote on the release,
> which will last for 72 hours. Consider this my (binding) +1.
>
> Airflow 1.10.0 RC 4 is available at:
>
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc4/ <
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc4/>
>
> apache-airflow-1.10.0rc4+incubating-source.tar.gz is a source release that
> comes with INSTALL instructions.
> apache-airflow-1.10.0rc4+incubating-bin.tar.gz is the binary Python
> "sdist"
> release.
>
> Public keys are available at:
>
> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>
> The amount of JIRAs fixed is over 700. Please have a look at the
> changelog.
> Since RC3 the following has been fixed:
>
> [AIRFLOW-2870] Use abstract TaskInstance for migration
> [AIRFLOW-2859] Implement own UtcDateTime
> [AIRFLOW-2140] Don't require kubernetes for the SparkSubmit hook
> [AIRFLOW-2869] Remove smart quote from default config
> [AIRFLOW-2857] Fix Read the Docs env
>
> Please note that the version number excludes the `rcX` string as well
> as the "+incubating" string, so it's now simply 1.10.0. This will allow us
> to rename the artifact without modifying the artifact checksums when we
> actually release.
>
> WARNING: Due to licensing requirements you will need to set
>  SLUGIFY_USES_TEXT_UNIDECODE=yes in your environment when
> installing or upgrading. We will try to remove this requirement for the
> next release.
>
> Cheers,
> Bolke