Sometimes mise installs precompiled linux-musl Python on linux-gnu systems

[egnor]

Describe the bug
On some Linux glibc systems, when installing a precompiled Python binary, mise will choose the statically linked linux-musl version instead of the dynamically linked linux-gnu version from the indygreg repository. The resulting binary works more-or-less but has a lot of quirks and incompatibilities since the world assumes that linux == glibc:

• poetry not compatible with precompiled python #1743
• [bug]: venv creation fails on linux-musl builds indygreg/python-build-standalone#261
• unable to install poetry indygreg/python-build-standalone#262

etc.

To Reproduce
This happens on some systems but not others! On a glibc system (normal desktop Linux, e.g. Ubuntu):

1. Install mise
2. mise settings set python_compile false
3. mise install python
4. ldd $(which python)

Expected behavior
Some output like this

% ldd $(which python)
	linux-vdso.so.1 (0x00007ffcdb3dc000)
	/home/egnor/.local/share/mise/installs/python/latest/bin/../lib/libpython3.12.so.1.0 (0x00007e126e000000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007e126f8d8000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007e126f8d3000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007e126f8ce000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007e126df15000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007e126f8c7000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007e126dc00000)
	/lib64/ld-linux-x86-64.so.2 (0x00007e126f8f3000)

Actual behavior

% ldd $(which python)
	not a dynamic executable

(because the Python binary is statically linked)

mise doctor output in BAD case

version: 2024.5.17 linux-x64 (ef63bec 2024-05-18)
activated: yes
shims_on_path: no

build_info: 
  Target: x86_64-unknown-linux-gnu
  Features: DEFAULT, NATIVE_TLS, OPENSSL
  Built: Sat, 18 May 2024 21:08:21 +0000
  Rust Version: rustc 1.78.0 (9b00956e5 2024-04-29)
  Profile: release

shell: 
  /bin/zsh
  zsh 5.9 (x86_64-ubuntu-linux-gnu)

dirs: 
  data: ~/.local/share/mise
  config: ~/.config/mise
  cache: ~/.cache/mise
  state: ~/.local/state/mise
  shims: ~/.local/share/mise/shims

config_files: 
  ~/.config/mise/config.toml

backends: 
  cargo
  go
  npm
  pipx
  ubi

plugins: 
  arduino-cli  https://github.com/egnor/asdf-arduino-cli.git#65fc0c8
  bun          (core)
  deno         (core)
  erlang       (core)
  go           (core)
  java         (core)
  node         (core)
  python       (core)
  ruby         (core)

toolset: 
  [email protected]   
  [email protected]  

env_vars: 
  MISE_SHELL=zsh

settings: 
  activate_aggressive = false
  all_compile = false
  always_keep_download = false
  always_keep_install = false
  asdf_compat = false
  cargo_binstall = true
  color = true
  disable_default_shorthands = false
  disable_tools = []
  experimental = false
  go_default_packages_file = "~/.default-go-packages"
  go_download_mirror = "https://dl.google.com/go"
  go_repo = "https://github.com/golang/go"
  go_set_gopath = false
  go_set_goroot = true
  go_skip_checksum = false
  jobs = 4
  legacy_version_file = true
  legacy_version_file_disable_tools = []
  node_compile = false
  not_found_auto_install = true
  paranoid = false
  plugin_autoupdate_last_check_duration = "7d"
  python_compile = false
  python_default_packages_file = "/home/egnor/.default-python-packages"
  python_pyenv_repo = "https://github.com/pyenv/pyenv.git"
  raw = false
  trusted_config_paths = []
  quiet = false
  verbose = false
  yes = false
  ci = false
  debug = false
  trace = false
  log_level = "info"
  python_venv_auto_create = false

  [status]
  missing_tools = "if_other_versions_installed"
  show_env = false
  show_tools = false

No warnings found
No problems found

mise doctor output in GOOD case

version: 2024.5.17 linux-x64 (ef63bec 2024-05-18)
activated: yes
shims_on_path: no

build_info: 
  Target: x86_64-unknown-linux-gnu
  Features: DEFAULT, NATIVE_TLS, OPENSSL
  Built: Sat, 18 May 2024 21:08:21 +0000
  Rust Version: rustc 1.78.0 (9b00956e5 2024-04-29)
  Profile: release

shell: 
  /bin/zsh
  zsh 5.9 (x86_64-ubuntu-linux-gnu)

dirs: 
  data: ~/.local/share/mise
  config: ~/.config/mise
  cache: ~/.cache/mise
  state: ~/.local/state/mise
  shims: ~/.local/share/mise/shims

config_files: 
  ~/.config/mise/config.toml

backends: 
  cargo
  go
  npm
  pipx
  ubi

plugins: 
  arduino-cli  https://github.com/egnor/asdf-arduino-cli.git#66bf3b6
  bun          (core)
  deno         (core)
  erlang       (core)
  go           (core)
  java         (core)
  node         (core)
  poetry       https://github.com/mise-plugins/mise-poetry.git#431c335
  python       (core)
  ruby         (core)

toolset: 
  [email protected]   
  [email protected]  

env_vars: 
  MISE_SHELL=zsh

settings: 
  activate_aggressive = false
  all_compile = false
  always_keep_download = false
  always_keep_install = false
  asdf_compat = false
  cargo_binstall = true
  color = true
  disable_default_shorthands = false
  disable_tools = []
  experimental = false
  go_default_packages_file = "~/.default-go-packages"
  go_download_mirror = "https://dl.google.com/go"
  go_repo = "https://github.com/golang/go"
  go_set_gopath = false
  go_set_goroot = true
  go_skip_checksum = false
  jobs = 4
  legacy_version_file = true
  legacy_version_file_disable_tools = []
  node_compile = false
  not_found_auto_install = true
  paranoid = false
  plugin_autoupdate_last_check_duration = "7d"
  python_compile = false
  python_default_packages_file = "/home/egnor/.default-python-packages"
  python_pyenv_repo = "https://github.com/pyenv/pyenv.git"
  raw = false
  trusted_config_paths = []
  quiet = false
  verbose = false
  yes = false
  ci = false
  debug = false
  trace = false
  log_level = "info"
  python_venv_auto_create = false

  [status]
  missing_tools = "if_other_versions_installed"
  show_env = false
  show_tools = false

No warnings found
No problems found

Note, basically identical. (Pretty sure the poetry plugin is incidental, that was from me trying to repro that bug...)

Additional context

It APPEARS that the builtin python plugin selects linux-musl vs linux-gnu based on whether the mise binary itself is linked against musl or glibc. That in turn is chosen when mise is installed, e.g. by the script returned from https://mise.run/, which currently has this logic to detect musl:

	if type ldd >/dev/null 2>/dev/null; then
		libc=$(ldd /bin/ls | grep 'musl' | head -1 | cut -d ' ' -f1)
		if [ -n "$libc" ]; then
			musl="-musl"
		fi
	fi

In fact on some (glibc) systems I'd seen mise installed as linux-musl (per mise doctor), BUT re-running the installer brought in a linux-gnu mise binary, BUT even so re-installing python still brought a linux-musl python binary.

[egnor]

Aha, reading the python.rs code, it caches the "matching" precompiled versions. After I blew away the mise cache and re-installed python I got a linux-gnu one.

So my belief is currently:

• linux-gnu mise installs linux-gnu python
• linux-musl mise installs linux-musl python
• HOWEVER that decision is cached and thus "sticky", so reinstalling mise doesn't change things unless you clear the cache
• the CURRENT https://mise.run/ script SEEMS to correctly install linux-gnu mise on linux-gnu systems, maybe that's unreliable, maybe that changed??

@yozachar you might try

1. curl https://mise.run/ | sh # reinstall mise
2. mise cache clear
3. mise uninstall python --all
4. mise install # or mise use python or whatever

... and see if you can create virtualenvs, install poetry, etc?

[jdx]

I'm going to see about keying the cache keys to include the target and maybe a few other things

[egnor]

Ideally, just cache the downloaded file, not any processing thereof?

(Also, DID the https://mise.run/ logic for -musl selection change at some point recently?)

[jdx]

it's better for the logic to be generic across all tools not just something for python. I think there was a change around that somewhat recently.

[egnor]

Aha yes: this bug is basically #1716, which was fixed by #1717, BUT cache persistence caused the old bad value to be "sticky" which is what bit myself and possibly @yozachar, so #2141 should prevent that going forward.