Incorrect stdlib path for relocated Python installs, resulting in ModuleNotFoundError: No module named 'encodings'

[edmorley]

Summary

uWSGI calculates the Python stdlib location incorrectly for Python installs that have been relocated, resulting in the server failing to start due to:

Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

This appears to be because uWSGI is using the wrong sysconfig APIs to calculate Python home and the location of the standard library - and so uWSGI attempts to use them from the hardcoded compilation time paths, rather than the new Python location reported by sys.prefix and other sysconfig APIs.

Steps to reproduce

1. mkdir uwsgi-testcase && cd $_
2. touch Dockerfile app.py
3. Edit Dockerfile and app.py to have the contents listed below
4. docker build -t uwsgi-test .
5. docker run --rm -it uwsgi-test
6. docker run --rm -it uwsgi-test python -c 'import pprint, sys, sysconfig; print("sys.prefix:", sys.prefix); print("sys.path:", pprint.pformat(sys.path)); print("sysconfig.get_paths():", pprint.pformat(sysconfig.get_paths()))'

# Dockerfile

FROM heroku/heroku:22-build

RUN mkdir -p /foo/python \
  && curl -f https://heroku-buildpack-python.s3.us-east-1.amazonaws.com/heroku-22/runtimes/python-3.11.2.tar.gz \
  | tar -zxC /foo/python

ENV PATH="/foo/python/bin:/root/.local/bin:${PATH}"
ENV CPATH="/foo/python/include/python3.11:${CPATH}"
ENV LD_LIBRARY_PATH="/foo/python/lib:${LD_LIBRARY_PATH}"
ENV LIBRARY_PATH="/foo/python/lib:${LIBRARY_PATH}"

RUN python -m ensurepip --default-pip
RUN pip install --user uwsgi==2.0.21
RUN uwsgi --version

WORKDIR /workspace
COPY app.py .

CMD uwsgi --http :9090 --wsgi-file app.py

# app.py

def application(env, start_response):
    start_response('200 OK', [('Content-Type','text/html')])
    return [b"Hello World"]

Expected

• The Python path configuration section of the uWSGI server output lists the correct locations for the Python installation (paths under /foo/python) that match what sys.prefix and sysconfig.get_paths() report.
• The uWSGI server starts successfully.

Actual

• The Python path configuration section of the uWSGI server output lists the wrong locations for the Python installation (paths under /app/.heroku/python) that don't match what sys.prefix and sysconfig.get_paths() report.
• The uWSGI server exits with an error trying to import the Python stdlib.
• In order to get the uWSGI server to boot, the PYTHONHOME env var has to be set to override uWSGI's incorrect location. For example, by running docker run --rm -it --env PYTHONHOME='/foo/python' uwsgi-test.

$ docker run --rm -it uwsgi-test
*** Starting uWSGI 2.0.21 (64bit) on [Tue Mar 21 12:20:13 2023] ***
compiled with version: 11.3.0 on 21 March 2023 12:14:36
os: Linux-5.15.49-linuxkit #1 SMP PREEMPT Tue Sep 13 07:51:32 UTC 2022
nodename: 25066dc0085b
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 6
current working directory: /workspace
detected binary path: /root/.local/bin/uwsgi
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
*** WARNING: you are running uWSGI without its master process manager ***
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on :9090 fd 6
spawned uWSGI http 1 (pid: 8)
uwsgi socket 0 bound to TCP address 127.0.0.1:44037 (port auto-assigned) fd 5
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) *** 
Python version: 3.11.2 (main, Feb  8 2023, 12:54:20) [GCC 11.3.0]
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/root/.local/bin/uwsgi'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/app/.heroku/python/lib/python3.11'
  sys._base_executable = '/root/.local/bin/uwsgi'
  sys.base_prefix = '/app/.heroku/python'
  sys.base_exec_prefix = '/app/.heroku/python'
  sys.platlibdir = 'lib'
  sys.executable = '/root/.local/bin/uwsgi'
  sys.prefix = '/app/.heroku/python'
  sys.exec_prefix = '/app/.heroku/python'
  sys.path = [
    '/app/.heroku/python/lib/python311.zip',
    '/app/.heroku/python/lib/python3.11',
    '/app/.heroku/python/lib/python3.11/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007ffffce99bc0 (most recent call first):
  <no Python frame>

In comparison, Python's stdlib has the correct paths:

$ docker run --rm -it uwsgi-test python -c 'import pprint, sys, sysconfig; print("sys.prefix:", sys.prefix); print("sys.path:", pprint.pformat(sys.path)); print("sysconfig.get_paths():", pprint.pformat(sysconfig.get_paths()))'
sys.prefix: /foo/python
sys.path: ['',
 '/foo/python/lib/python311.zip',
 '/foo/python/lib/python3.11',
 '/foo/python/lib/python3.11/lib-dynload',
 '/root/.local/lib/python3.11/site-packages',
 '/foo/python/lib/python3.11/site-packages']
sysconfig.get_paths(): {'data': '/foo/python',
 'include': '/foo/python/include/python3.11',
 'platinclude': '/foo/python/include/python3.11',
 'platlib': '/foo/python/lib/python3.11/site-packages',
 'platstdlib': '/foo/python/lib/python3.11',
 'purelib': '/foo/python/lib/python3.11/site-packages',
 'scripts': '/foo/python/bin',
 'stdlib': '/foo/python/lib/python3.11'}

Versions

• uWSGI v2.0.21
• Python 3.11.2 (it also reproduces on older Python, eg 3.9.16)
• Ubuntu 22.04 (though somewhat irrelevant, given it's a custom build of Python and not the distro version)

Notes

• The Python runtime archive used in the testcase above has been relocated. ie: It was compiled for one location, and then is run from another by end users in a separate Docker image. We have no control over what this final location is, and cannot compile at time of use since that would take too long. (The original context for this issue is as part of a Cloud Native Buildpack build: Error running locally built image with pack: Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding heroku/buildpacks-python#18. To keep the testcase simple I've used a standard Dockerfile instead.)
• Relocating the Python installation is a fully supported scenario by upstream CPython, as demonstrated by the fact that sys.prefix and sysconfig.get_paths() report the correct locations. How Python determines the Python location is documented here.
• It seems the issue is that uWSGI is using the build time env vars from sysconfig (eg LIBS and SYSLIBS) which are hardcoded by Python at time of build, and so don't change when the Python installation is relocated. To resolve this issue, it seems these usages should be switched to one of the other sysconfig APIs instead. See:

  • uwsgi/plugins/python/uwsgiplugin.py

    Line 40 in 73efb01

    LIBS = sysconfig.get_config_var('LIBS').split() + sysconfig.get_config_var('SYSLIBS').split()

  • https://docs.python.org/3.11/library/sysconfig.html#installation-paths

[edmorley]

Something else I've just discovered - this issue only affects uWSGI when it was installed into the user site-packages using using pip install --user. If uWSGI was installed into the system site-packages or else into a virtualenv's site-packages, then the error doesn't occur.