git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[DISCUSS] Dropping support for CentOS 5 / RHEL5 in Python packages


hi folks,

Surfacing a JIRA discussion ([4]) to the mailing list for discussion.

The manylinux1 ABI was developed to provide a mechanism for portable
Python packages with pre-compiled binary extensions supporting C and
C++, including C++11, on a wide variety of Linux distributions without
need for distribution-specific packages. This is accomplished using
RedHat's devtoolset-2, which performs selecting static linking of
symbols from libstdc++ that cause ABI conflicts when used on systems
with older standard libraries.

The base image for producing these binaries is specified in a Dockerfile [1].

The problem that we are having is that some C++ libraries, notably
Google's Abseil C++ library, require a version of glibc that is too
new for RHEL5. By building with CentOS6 / RHEL6 as the base image, we
would get a new enough glibc (version 2.12). But building against
glibc 2.12 would leave behind the RHEL5 folks.

There is the in-discussion manylinux2010 standard uses RHEL6 as a base
standard, but it is not yet finalized or in production.

Some modern C++ projects shipping to Python have already left behind
the manylinux1 standard even though their Python binaries claim to
implement the standard. Both PyTorch and TensorFlow are tagged as
manylinux1 although they have a different ABI. See [2] for example and
[3]

In my view there are two paths forward, neither perfect:

1) Stick with the manylinux1 ABI and do not use thirdparty libraries
requiring newer glibc
2) "Cheat" on manylinux1 by using centos6 instead of centos5 as the
base image for the wheel builds. This is what PyTorch is doing

Since centos5 / RHEL5 are already past EOL those would be the primary
casualties, but I'm not sure how many users would be affected. My
guess is that they represent a small minority of our users at this
point. RedHat is offering extended support for RHEL5 through end of
2020 but those are probably fairly exceptional cases and unlikely
(IMHO) to be working on the bleeding edge of Python data engineering.

Personally I would like to go with Option 2 and hope that this
particular Python packaging gets sorted out in the next 12-24 months
as we've already suffered problems due to TensorFlow and PyTorch's
non-conformity with the manylinux1 ABI.

Interested in the opinions of others.

- Wes

[1]: https://github.com/pypa/manylinux/blob/master/docker/Dockerfile-x86_64
[2]: https://github.com/NVIDIA/nvidia-docker/issues/348#issuecomment-288875848
[3]: https://github.com/pypa/manylinux/issues/96
[4]: https://issues.apache.org/jira/browse/ARROW-2461