Support manylinux1 wheels

[rouge8]

Pex doesn't seem to support manylinux1 wheels:

$ pex --version && pex --no-build -vvv numpy
pex 1.1.11
pex: crawling link i=0 link=Link('file:///home/andy/.pex/build') follow_links=False
pex: crawling link i=0 link=Link('https://pypi.python.org/simple/numpy/') follow_links=False
Could not satisfy all requirements for numpy:
    numpy
$ pip install --only-binary :all: numpy
Collecting numpy
  Downloading numpy-1.11.1-cp27-cp27mu-manylinux1_x86_64.whl (15.3MB)
    100% |████████████████████████████████| 15.3MB 102kB/s
Installing collected packages: numpy
Successfully installed numpy-1.11.1

This was mentioned briefly in #250, but that was closed.

[rouge8]

PEP 513 has an algorithm for detecting manylinux support, although pip has modified it slightly to deal with weird distributions: pypa/pip#3497, pypa/pip#3590

[rouge8]

I've been thinking about tackling this, but I can't decide how/whether to handle manylinux compatibility at buildtime. At runtime, you can just use the PEP513 (or modified pip) algorithm, but that won't work for building a Linux PEX on an OS X host. The best I've thought of is using _manylinux.manylinux1_compatible and only applying that portion of the compatibility check:

To handle potential future incompatibilities, we standardize a mechanism for a Python distributor to signal that a particular Python install definitely is or is not compatible with manylinux1 : this is done by installing a module named _manylinux , and setting its manylinux1_compatible attribute. We do not propose adding any such module to the standard library -- this is merely a well-known name by which distributors and installation tools can rendezvous. However, if a distributor does add this module, they should add it to the standard library rather than to a site-packages/ directory, because the standard library is inherited by virtualenvs (which we want), and site-packages/ in general is not.

Similar to pypa/pip#3689.

Or is it reasonable to assume manylinux1 compatibility when building?

[kwlzn]

how about making it explicitly user controlled at pex creation time - e.g. with a feature flag like --manylinux to enable resolving against manylinux artifacts (when --platform contains the string linux)?

then if the user wants to build a manylinux compatible pex for running on a separate host, they'd just pass this flag - and then separately be gated by the runtime compatibility checking when running the pex on the separate host where the constraint actually matters.

[rouge8]

Sounds reasonable. --manylinux/--no-manylinux maybe and default to --manylinux to match the behavior of pip.

[ariscn]

+1, more and more packages are providing only manylinux1 wheels. It would be great if pex could pull those in instead of requiring linux-x86_64 artifacts.

[bragan]

another +1 on this. We have run into this issue with libsass which is only providing manylinux1 so we had to pin the last version... any update on manylinux pex support?

[rouge8]

Unfortunately I'm not using PEX at my current job and haven't had much time to spend on this. 😞

[sholsapp]

+1 if/when you get to it :) This 🔥'd me.

[augray]

+1

[fvinas]

+1 ;-)

[evanj]

I found a disgusting hack that "works" for me. At the very least, when I build a pex using some manylinux1 wheels, it will allow it to be run.

In package.py, in the function platform_iterator(...), I added the following "hack" which treats all Linux distributions as manylinux1 compatible. Strictly speaking this is not true, and Pip has a bunch of rules in pip.pep425tags that Pex should probably be reusing. However, this was enough to get it to work for me:

  def platform_iterator(cls, platform):
    """Iterate over all compatible platform tags of a supplied platform tag.

       :param platform: the platform tag to iterate over
    """
    if cls.is_macosx_platform(platform):
      for plat in cls.iter_compatible_osx_platforms(platform):
        yield plat
    elif platform == 'linux_x86_64':
      yield 'manylinux1_x86_64'
      yield platform
    else:
      yield platform

[anthrotype]

What's the status of this? There seems to be two competing PRs #316 and #399, both in an unmergeable state. It would be great if pex also supported manylinux1. Thank you!

[kwlzn]

the final implementation is now reviewable here: #480

[kwlzn]

this is now available as of pex 1.4.0 (on pypi).

pex will now resolve against manylinux dists both at build and runtime (implicitly, by default) when targeting linux platforms, with no special platform targeting required:

[omerta ~]$ pex --version
pex 1.4.0
[omerta ~]$ pex --disable-cache --no-build --platform=linux-x86_64 msgpack_python -o /tmp/msgpack.pex
[omerta ~]$ unzip -l /tmp/msgpack.pex | grep manylinux | head -1
warning [/tmp/msgpack.pex]:  25 extra bytes at beginning or within zipfile
  (attempting to process anyway)
     1385  05-15-2018 16:56   .deps/msgpack_python-0.4.7-cp27-cp27m-manylinux1_x86_64.whl/msgpack/__init__.py

if you'd like to disable manylinux resolves (1.3.x and earlier behavior), you can pass --no-manylinux:

[omerta ~]$ pex --disable-cache --no-build --platform=linux-x86_64 --no-manylinux msgpack_python -o /tmp/msgpack.pex
Could not satisfy all requirements for msgpack_python:
    msgpack_python

you can also do more granular platform/tag targeting via an extended --platform mode:

[omerta ~]$ pex --disable-cache --no-build --platform=linux-x86_64-cp-27-cp27mu msgpack_python -o /tmp/msgpack.pex
[omerta ~]$ unzip -l /tmp/msgpack.pex | grep manylinux | head -1
warning [/tmp/msgpack.pex]:  25 extra bytes at beginning or within zipfile
  (attempting to process anyway)
     1385  05-15-2018 16:58   .deps/msgpack_python-0.4.7-cp27-cp27mu-manylinux1_x86_64.whl/msgpack/__init__.py

...which still supports multi-platform usage:

[omerta ~]$ pex --disable-cache --no-build --platform=linux-x86_64-cp-27-cp27mu --platform=linux-x86_64-cp-27-cp27m msgpack_python -o /tmp/msgpack.pex
[omerta ~]$ unzip -l /tmp/msgpack.pex | grep "cp27m-" | head -1
warning [/tmp/msgpack.pex]:  25 extra bytes at beginning or within zipfile
  (attempting to process anyway)
     1385  05-15-2018 16:59   .deps/msgpack_python-0.4.7-cp27-cp27m-manylinux1_x86_64.whl/msgpack/__init__.py
[omerta ~]$ unzip -l /tmp/msgpack.pex | grep "cp27mu-" | head -1
warning [/tmp/msgpack.pex]:  25 extra bytes at beginning or within zipfile
  (attempting to process anyway)
     1385  05-15-2018 16:59   .deps/msgpack_python-0.4.7-cp27-cp27mu-manylinux1_x86_64.whl/msgpack/__init__.py
[omerta ~]$ 

please take a moment to kick the tires with pex 1.4.0 and open issues for any bugs or problems that you might discover. enjoy!

[alecbz]

This seems to still not work for me. When I download a manylinux wheel, pex can't seem to find it, but if I just rename manylinux1 to linux, it works.

When using the manylinux wheel for psycopg2==2.7.4 named as psycopg2-2.7.4-cp27-cp27mu-manylinux1_x86_64.whl, my build eventually fails with:

Exception message: Package SourcePackage(u'file:///Users/alec/.cache/pants/python_cache/requirements/CPython-2.7.10/psycopg2-2.7.4.tar.gz') is not translateable by ChainedTranslator(Wh
eelTranslator, EggTranslator, SourceTranslator) 

(the mechanics of what exactly is happening here are still a mystery to me, but I'm guessing it's something like: no suitable wheel is found, so it starts trying to build from the source tar.gz, and can't because I'm not on Linux)

but when I then clear my ~/.cache/pants/python_cache/requirements and rename the wheel to psycopg2-2.7.4-cp27-cp27mu-linux_x86_64.whl, the build works.

---

My pex --version is 1.4.5, but a wrinkle is that I'm using pex via pants, and I'm not actually sure if the version of pex on my $PATH is the same version pants ends up using.

[stuhood]

My pex --version is 1.4.5, but a wrinkle is that I'm using pex via pants, and I'm not actually sure if the version of pex on my $PATH is the same version pants ends up using.

Pants resolves its own version of pex. Can find-in-page for pex here: https://www.pantsbuild.org/notes-master.html ... to see when various versions were incorporated.

[alecbz]

Ah, ok. Couldn't find it from that page directly, but I'm on pants 1.7.0, and judging from

https://github.com/pantsbuild/pants/blob/1.7.x/3rdparty/python/requirements.txt#L17

I guess I'm on pex 1.3.2 still.