Installing packages with Salt-Pip fails: JSONDecodeError

[vps-eric]

Installing packages with salt-pip (Salt 3006.0) fails with a JSONDecodeError and what appears to be some recursive behavior.

To reproduce, set up a fresh Almalinux 8.7 box (I use Vagrant on QEMU) and manually install the Salt 3006.0 minion. Add a symlink according to #114:

ln -s /opt/saltstack/salt/bin/python3 /usr/bin/python3

Then, attempt to install pycryptodome (or any other package in my experience):

/opt/saltstack/salt/salt-pip install pycryptodome

Stack Trace

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/saltstack/salt/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/cli/main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/cli/autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/cli/main_parser.py", line 9, in <module>
    from pip._internal.build_env import get_runnable_pip
  File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/build_env.py", line 20, in <module>
    from pip._internal.cli.spinners import open_spinner
  File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/cli/spinners.py", line 9, in <module>
    from pip._internal.utils.logging import get_indentation
  File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/utils/logging.py", line 29, in <module>
    from pip._internal.utils.misc import ensure_dir
  File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/utils/misc.py", line 42, in <module>
    from pip._internal.locations import get_major_minor_version
  File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/locations/__init__.py", line 14, in <module>
    from . import _sysconfig
  File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/locations/_sysconfig.py", line 11, in <module>
    from .base import change_root, get_major_minor_version, is_osx_framework
  File "/opt/saltstack/salt/lib/python3.10/site-packages/pip/_internal/locations/base.py", line 16, in <module>
    site_packages: typing.Optional[str] = sysconfig.get_path("purelib")
  File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 572, in get_path
    return get_paths(scheme, vars, expand)[name]
  File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 167, in wrapped
    paths = func(scheme=scheme, vars=vars, expand=expand)
  File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 562, in get_paths
    return _expand_vars(scheme, vars)
  File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 218, in _expand_vars
    _extend_dict(vars, get_config_vars())
  File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 131, in wrapped
    _CONFIG_VARS = func()
  File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 640, in get_config_vars
    srcdir = os.path.dirname(get_makefile_filename())
  File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 399, in get_makefile_filename
    return os.path.join(get_path('stdlib'), config_dir_name, 'Makefile')
  File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 572, in get_path
    return get_paths(scheme, vars, expand)[name]
  File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 167, in wrapped
    paths = func(scheme=scheme, vars=vars, expand=expand)
  File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 562, in get_paths
    return _expand_vars(scheme, vars)
  File "/opt/saltstack/salt/lib/python3.10/sysconfig.py", line 218, in _expand_vars
    _extend_dict(vars, get_config_vars())
  File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 140, in wrapped
    _SYSTEM_CONFIG_VARS = json.loads(p.stdout[:-1])
  File "/opt/saltstack/salt/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/opt/saltstack/salt/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/saltstack/salt/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The JSONDecodeError is due to get_config_vars_wrapper(...) parsing the stdout of a subprocess Python call to sysconfig: in my environment, calling sysconfig.get_config_vars() actually prints nothing (stdout empty), causing JSON decoding to fail. (Side note: did Rube Goldberg write this function? I'm sure there's a reason for the way this is implemented; could someone elaborate?)

Digging deeper into sysconfig.get_config_vars() reveals some apparently recursive behavior. The following is what I have observed in my environment when calling sysconfig.get_config_vars() in the same manner as get_config_vars_wrapper() from Salt 3006.0's Python 3.10 REPL:

1. sysconfig.get_config_vars() is called without arguments. _CONFIG_VARS is None, the os.name is posix and _PYTHON_BUILD is falsy. Execution continues to get_makefile_filename().
2. Execution continues to a call to get_path('stdlib').
3. get_paths('posix_prefix', None, True) is called. It in turn calls _expand_vars('posix_prefix', None).
4. _expand_vars calls _extend_dict(None, get_config_vars()). Notice that get_config_vars() is now being recursively called.
5. At this point in practice, the wrapper is called instead of get_config_vars. The wrapper, again, calls sysconfig.get_config_vars(None) within a new Python subprocess, and this time _CONFIG_VARS has a bunch (722 for me) of variables, which it returns.
6. For some reason I haven't wrapped my mind around, the now non-empty variables result is not returned from the wrapper back to _extend_dict. In fact, _extend_dict is never called. Instead, the wrapper recursively calls itself. I've seen it do so up to 1,024 times, and have heard from another user with this problem that it is accompanied by a spike in memory and crash of the Salt minion and master.

Interestingly, this behavior seems to only occur when using Salt's bundled Python at /usr/bin/python3. Compiling Python 3.10.11 from source and adding the appropriate symlink for that scenario (ln -s /usr/local/bin/python3 /usr/bin/python3) does not present this problem.

[vps-eric]

See #114; Python should be installed on the system.