Issues with wheel dependencies with pex

[vpal]

I'm trying to create a pex package from my python package with the following dependencies:

lockfile
gitpython
boto3
awacs
troposphere
pymysql
bcrypt

When I just create a virtaualenv on my machine these dependencies install without issues and with using the wheel packages instead of compiling.

If I try to build the pex package I get compilation errors like this:

**** Failed to install bcrypt-3.1.1 (caused by: NonZeroExit("received exit code 1 during execution of `['/tmp/tmp.zgQZB7JLtU/bin/python2', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpGwiZus']` while trying to execute `['/tmp/tmp.zgQZB7JLtU/bin/python2', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpGwiZus']`",)
):
stdout:

stderr:
c/_cffi_backend.c:2:20: fatal error: Python.h: No such file or directory

If I add the --no-build I get this error:

----- Running pex -----
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x7f6c450eaa50>
pex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x7f6c450ea290>        
pex: crawling link i=0 link=Link('https://pypi.python.org/simple/setuptools/') follow_links=False
pex: Building pex :: Resolving interpreter :: Setting up interpreter /tmp/tmp.QkBIt95fNM/bin/python2 :: Interpreter cache resolving setuptools<20.11,>=2.2 :: Fetching SourcePackage(u'https://pypi.python.org/packpex: Building pex :: Resolving interpreter :: Setting up interpreter /tmp/tmp.QkBIt95fNM/bin/python2 :: Interpreter cache resolving setuptools<20.11,>=2.2 :: Fetching SourcePackage(u'https://pypi.python.org/packages/d3/38/3f947884c009fc65c4d9f60ff154230be6226ac2215a30e631e155418a30/setuptools-12.3.tar.gz#md5=67614b6d560fa4f240e99cd553ec7f32') :: Fetching https://pypi.python.org/packages/d3/38/3f947884c009fc65c4d9f60ff1pex: Validated setuptools-12.3.tar.gz (md5=67614b6d560fa4f240e99cd553ec7f32)                                                                                                                                                                                                                                                                                                                                                                                                                                                        
pex: Validated setuptools-12.3.tar.gz (md5=67614b6d560fa4f240e99cd553ec7f32)
pex: Building pex :: Resolving interpreter :: Setting up interpreter /tmp/tmp.QkBIt95fNM/bin/python2 :: Interpreter cache resolving setuptools<20.11,>=2.2 :: Installing SourcePackage(u'https://pypi.python.org/papex: Building pex :: Resolving interpreter :: Setting up interpreter /tmp/tmp.QkBIt95fNM/bin/python2 :: Interpreter cache resolving setuptools<20.11,>=2.2 :: Installing SourcePackage(u'https://pypi.python.org/papex: Constructed RequestsContext context <pex.http.RequestsContext object at 0x7f6c450ea9d0>                                                                                                                                                                                                                                                                                           
pex: crawling link i=0 link=Link('https://pypi.python.org/simple/wheel/') follow_links=False
pex: Building pex :: Resolving interpreter :: Setting up interpreter /tmp/tmp.QkBIt95fNM/bin/python2 :: Interpreter cache resolving wheel<0.30.0,>=0.26.0 :: Fetching SourcePackage(u'https://pypi.python.org/packapex: Building pex :: Resolving interpreter :: Setting up interpreter /tmp/tmp.QkBIt95fNM/bin/python2 :: Interpreter cache resolving wheel<0.30.0,>=0.26.0 :: Fetching SourcePackage(u'https://pypi.python.org/packages/8f/c6/c524fe41283b2820f85a8e38cedd19ebb6454365281549dcf98ae32feb85/wheel-0.28.0.tar.gz#md5=6d6b7e87dd744c6fac77ca9b43479207') :: Fetching https://pypi.python.org/packages/8f/c6/c524fe41283b2820f85a8e38cedd19pex: Validated wheel-0.28.0.tar.gz (md5=6d6b7e87dd744c6fac77ca9b43479207)                                                                                                                                                                                                                                                                                                                                                                                                                                                    
pex: Validated wheel-0.28.0.tar.gz (md5=6d6b7e87dd744c6fac77ca9b43479207)
pex: Building pex :: Resolving interpreter :: Setting up interpreter /tmp/tmp.QkBIt95fNM/bin/python2 :: Interpreter cache resolving wheel<0.30.0,>=0.26.0 :: Installing SourcePackage(u'https://pypi.python.org/pacpex: Building pex :: Resolving interpreter :: Setting up interpreter /tmp/tmp.QkBIt95fNM/bin/python2 :: Interpreter cache resolving wheel<0.30.0,>=0.26.0 :: Installing SourcePackage(u'https://pypi.python.org/pacpex: Building pex :: Resolving distributions                                                                                                                                                                       pex: crawling link i=0 link=Link('https://pypi.python.org/simple/lockfile/') follow_links=False                                                                 
pex: crawling link i=0 link=Link('https://pypi.python.org/simple/gitpython/') follow_links=False
pex: crawling link i=0 link=Link('https://pypi.python.org/simple/boto3/') follow_links=False
pex: crawling link i=0 link=Link('https://pypi.python.org/simple/awacs/') follow_links=False
Could not satisfy all requirements for awacs:
    awacs

How can I make pex build the pex file without compiling anything, but using the wheel distributions.

I have tried most of the command line options in many different combinations like providing the platfom, etc. and did a lot of googling without any luck.

I think that the two packages that pex wants to compile are cffi and bcrypt.

Would be thankful to get some help here.

Thanks

[kwlzn]

How can I make pex build the pex file without compiling anything, but using the wheel distributions.

well, the first step to getting pex to use wheel distributions sans a build step is to actually have prebuilt wheels available. afaict, at least for awacs there are no wheels on pypi. additionally, it's not clear which platform you're on - but for packages like bcrypt on linux there are only manylinux wheels and pex does not yet support manylinux (see #281) - which is probably why it's attempting to build cffi and bcrypt. this is only a platform naming distinction, so simply renaming a manylinux platform wheel to linux-x86_64 is a suitable workaround for the time being.

in most cases at Twitter, we use a separate process for building wheels vs resolving them. for you, this would look approximately like this:

[illuminati show]$ cat requirements.txt 
lockfile
gitpython
boto3
awacs
troposphere
pymysql
bcrypt
[illuminati show]$ mkdir wheels
[illuminati show]$ pip wheel --no-cache-dir --wheel-dir=./wheels -r requirements.txt
Collecting lockfile (from -r requirements.txt (line 1))
  Downloading lockfile-0.12.2-py2.py3-none-any.whl
  Saved ./wheels/lockfile-0.12.2-py2.py3-none-any.whl
Collecting gitpython (from -r requirements.txt (line 2))
  Downloading GitPython-2.1.0-py2.py3-none-any.whl (441kB)
    100% |████████████████████████████████| 450kB 12.2MB/s 
  Saved ./wheels/GitPython-2.1.0-py2.py3-none-any.whl
Collecting boto3 (from -r requirements.txt (line 3))
  Downloading boto3-1.4.1-py2.py3-none-any.whl (121kB)
    100% |████████████████████████████████| 122kB 19.2MB/s 
  Saved ./wheels/boto3-1.4.1-py2.py3-none-any.whl
Collecting awacs (from -r requirements.txt (line 4))
  Downloading awacs-0.6.0.tar.gz
Collecting troposphere (from -r requirements.txt (line 5))
  Downloading troposphere-1.8.2.tar.gz (83kB)
    100% |████████████████████████████████| 92kB 20.5MB/s 
Collecting pymysql (from -r requirements.txt (line 6))
  Downloading PyMySQL-0.7.9-py2-none-any.whl (78kB)
    100% |████████████████████████████████| 81kB 28.1MB/s 
  Saved ./wheels/PyMySQL-0.7.9-py2-none-any.whl
Collecting bcrypt (from -r requirements.txt (line 7))
  Downloading bcrypt-3.1.1-cp27-cp27m-macosx_10_6_intel.whl (50kB)
    100% |████████████████████████████████| 51kB 27.5MB/s 
  Saved ./wheels/bcrypt-3.1.1-cp27-cp27m-macosx_10_6_intel.whl
Collecting gitdb2>=2.0.0 (from gitpython->-r requirements.txt (line 2))
  Downloading gitdb2-2.0.0-py2.py3-none-any.whl (63kB)
    100% |████████████████████████████████| 71kB 31.9MB/s 
  Saved ./wheels/gitdb2-2.0.0-py2.py3-none-any.whl
Collecting s3transfer<0.2.0,>=0.1.0 (from boto3->-r requirements.txt (line 3))
  Downloading s3transfer-0.1.9-py2.py3-none-any.whl (53kB)
    100% |████████████████████████████████| 61kB 39.5MB/s 
  Saved ./wheels/s3transfer-0.1.9-py2.py3-none-any.whl
Collecting jmespath<1.0.0,>=0.7.1 (from boto3->-r requirements.txt (line 3))
  Downloading jmespath-0.9.0-py2.py3-none-any.whl
  Saved ./wheels/jmespath-0.9.0-py2.py3-none-any.whl
Collecting botocore<1.5.0,>=1.4.1 (from boto3->-r requirements.txt (line 3))
  Downloading botocore-1.4.70-py2.py3-none-any.whl (2.8MB)
    100% |████████████████████████████████| 2.8MB 16.3MB/s 
  Saved ./wheels/botocore-1.4.70-py2.py3-none-any.whl
Collecting cffi>=1.1 (from bcrypt->-r requirements.txt (line 7))
  Downloading cffi-1.8.3-cp27-cp27m-macosx_10_10_intel.whl (213kB)
    100% |████████████████████████████████| 215kB 13.0MB/s 
  Saved ./wheels/cffi-1.8.3-cp27-cp27m-macosx_10_10_intel.whl
Collecting six>=1.4.1 (from bcrypt->-r requirements.txt (line 7))
  Downloading six-1.10.0-py2.py3-none-any.whl
  Saved ./wheels/six-1.10.0-py2.py3-none-any.whl
Collecting smmap2>=2.0.0 (from gitdb2>=2.0.0->gitpython->-r requirements.txt (line 2))
  Downloading smmap2-2.0.1-py2.py3-none-any.whl
  Saved ./wheels/smmap2-2.0.1-py2.py3-none-any.whl
Collecting futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" (from s3transfer<0.2.0,>=0.1.0->boto3->-r requirements.txt (line 3))
  Downloading futures-3.0.5-py2-none-any.whl
  Saved ./wheels/futures-3.0.5-py2-none-any.whl
Collecting docutils>=0.10 (from botocore<1.5.0,>=1.4.1->boto3->-r requirements.txt (line 3))
  Downloading docutils-0.12.tar.gz (1.6MB)
    100% |████████████████████████████████| 1.6MB 16.0MB/s 
Collecting python-dateutil<3.0.0,>=2.1 (from botocore<1.5.0,>=1.4.1->boto3->-r requirements.txt (line 3))
  Downloading python_dateutil-2.6.0-py2.py3-none-any.whl (194kB)
    100% |████████████████████████████████| 194kB 14.6MB/s 
  Saved ./wheels/python_dateutil-2.6.0-py2.py3-none-any.whl
Collecting pycparser (from cffi>=1.1->bcrypt->-r requirements.txt (line 7))
  Downloading pycparser-2.17.tar.gz (231kB)
    100% |████████████████████████████████| 235kB 13.5MB/s 
Skipping lockfile, due to already being wheel.
Skipping gitpython, due to already being wheel.
Skipping boto3, due to already being wheel.
Skipping pymysql, due to already being wheel.
Skipping bcrypt, due to already being wheel.
Skipping gitdb2, due to already being wheel.
Skipping s3transfer, due to already being wheel.
Skipping jmespath, due to already being wheel.
Skipping botocore, due to already being wheel.
Skipping cffi, due to already being wheel.
Skipping six, due to already being wheel.
Skipping smmap2, due to already being wheel.
Skipping futures, due to already being wheel.
Skipping python-dateutil, due to already being wheel.
Building wheels for collected packages: awacs, troposphere, docutils, pycparser
  Running setup.py bdist_wheel for awacs ... done
  Stored in directory: /private/tmp/show/wheels
  Running setup.py bdist_wheel for troposphere ... done
  Stored in directory: /private/tmp/show/wheels
  Running setup.py bdist_wheel for docutils ... done
  Stored in directory: /private/tmp/show/wheels
  Running setup.py bdist_wheel for pycparser ... done
  Stored in directory: /private/tmp/show/wheels
Successfully built awacs troposphere docutils pycparser

which would produce a directory of wheels from pypi satisfying the transitive deps from your requirements.txt. then, once you have this you can point pex directly at this directory of pre-built wheels like so:

[illuminati show]$ pex -r requirements.txt --repo=./wheels --no-pypi --no-build -o app.pex
[illuminati show]$ ./app.pex
Python 2.7.10 (default, Dec 16 2015, 14:09:45) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import cffi
>>> cffi.__file__
'/Users/kwilson/.pex/install/cffi-1.8.3-cp27-cp27m-macosx_10_10_intel.whl.31ff96353426795bc1748154ca6d94996f99289b/cffi-1.8.3-cp27-cp27m-macosx_10_10_intel.whl/cffi/__init__.py'
>>> 

or you can also host wheels on an http server and point --repo to a URL instead.

[vpal]

Thanks a lot for your help.
I have already been there before (try to download/build wheels into a directory) and tried that and got the error:

Could not satisfy all requirements for bcrypt:
    bcrypt

I finally now found the solution which is that you need to add the --platform=manylinux1_x86_64 option.
I was running pex on Linux and the target is also Linux.
And I assume that pex is trying something like linux-x86_64 for the platform, but the Linux wheels have the manylinux1_x86_64 in their name.
As this is handled by pip it would be nice if pex would also handle this or give some hints in the error message.

Thanks.

Edit:
1.)
And the --ignore-errors option is also needed because otherwise I get the following error:

Failed to execute PEX file, missing compatible dependencies for:
bcrypt
cffi

2.)
Even this doesn't help, because you get the following runtime error:

    import bcrypt
ImportError: No module named bcrypt

3.)
The only thing that solved the problem was actually renaming the wheel packages like this:

  for pkg in "${tmp_dir}/wheels/"*; do
    if [[ "${pkg}" == *"manylinux1_x86_64"* ]]; then
      mv "${pkg}" "${pkg/manylinux1_x86_64/linux_x86_64}"
    fi
  done

I don't know if I'm doing something wrong or if this is how this meant to be done, but it is working so that is definitely very good. :)

[kwlzn]

yup, as suggested above renaming the wheels from the manylinux platform type to linux-x86_64 is the best workaround right now.

specifying the platform at build time will definitely get the manylinux wheels into the pex, but there's also a runtime component that determines which platform you're running on and then utilizes the correct set of wheels - this is the main part that isn't currently manylinux compatible. #281 is open for that.

[vpal]

Thanks a lot, I'm very glad that I was able to find help here.