Support for full bundling of native dependencies / static linking

[dsvensson]

Pex is cool when it comes to pure Python modules, and cases where you have control over the target environment. This is for example not the case for code to be deployed at AWS Lambda for example, or if you want to share your shiny PyQt-based app with some friends.

It would be very nice if Pex had support for gathering of dependencies, and setting up the appropriate library paths etc for the application to be able to execute fully from within the Pex file based on the OS/architecture it's executed under.

Without the support there will just be more and more projects like this:
https://github.com/moesy/AWS-Lambda-ML-Microservice-Skeleton
https://github.com/Miserlou/lambda-packages

It might be that this is a deeper problem with Python, and how Python modules are built, but to me as a user, it's what I expect to see when reading the description:

"...and makes the deployment of Python applications as simple as cp. pex files may even include multiple platform-specific Python distributions, meaning that a single pex file can be portable across Linux and OS X..."

[kwlzn]

It might be that this is a deeper problem with Python...

yes, this is a much deeper problem with python packaging that pex has no real control over. additionally, there is no 100% uniform way to ensure compatible static linking for all native dependencies at build time without additional configuration and components such as docker and pre-built docker images, which would vastly complicate the pex runtime.

the way most production shops solve this is by pre-building known-to-work bdists and handing those to pex at resolve time, rather than letting pex resolve against sdists which invoke setup.py to compile native code from source. this essentially pushes the work and control over static linking and compatibility up into the layer above pex, which I think is much more reliable and desirable. this approach also allows you to compose pex files in way that would otherwise require cross-compilation. for example, with pre-built bdists you can build a pex file with dependencies for both OSX and Linux from just your e.g. OSX laptop.

for the general case of "one binary dependency that executes on as many linux platforms as possible" tho, I think what you basically want here is manylinux. theoretically, if you target manylinux bdists (many of which get published directly to pypi), those should provide better runtime compatibility guarantees across Linux platforms.

the issue tracking manylinux support in pex is #281, so will close this one as a duplicate of that - so would subscribe to that issue for updates.