Too many open files

[jonashaag]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

[project]
...
channels = ["conda-forge"]
platforms = ["linux-64", "linux-aarch64", "osx-arm64"]

[dependencies]
beautifulsoup4 = ">=4,<5"
chromadb = "<1"
coverage = ">=7.4.1,<7.5"
cyclonedx-bom = ">=3.11.7,<3.12"
datamodel-code-generator = ">=0.25.4,<0.26"
dateparser = ">=1.2.0,<1.3"
easyocr = ">=1.7.1,<1.8"
fastapi = ">=0.103.0,<0.104"
httpx = ">=0.26.0,<0.27"
ipython = ">=8.21.0,<8.22"
jupyter = ">=1.0.0,<1.1"
lxml = ">=5.1.0,<5.2"
openai = ">=1.35.3,<1.36"
openpyxl = ">=3.1.2,<3.2"
pandas = "2.*"
passlib = ">=1.7.4,<1.8"
pdbpp = ">=0.10.3,<0.11"
pdf2image = "1.16.*"
pixi-pycharm = ">=0.0.5,<0.1"
polars = ">=0.20.23,<0.21"
py-spy = ">=0.3.14,<0.4"
py7zr = ">=0.20.8,<0.21"
pydantic = ">=2.6.1,<2.7"
pyexcel = ">=0.7.0,<0.8"
pyexcel-xls = ">=0.6.2,<0.7"
pyexcel-xlsx = ">=0.6.0,<0.7"
pymongo = ">=4.6.1,<4.7"
pypdf = ">=4.0.2,<4.1"
pytesseract = ">=0.3.13,<0.4"
pytest = "<8" # https://github.com/TvoroG/pytest-lazy-fixture/issues/65
python = ">=3.12"
python-dotenv = ">=1.0.1,<1.1"
python-jose = ">=3.3.0,<3.4"
python-multipart = ">=0.0.7,<0.1"
sentence-transformers = ">=2.6.1,<2.7"
tabulate = ">=0.9.0,<0.10"
tesseract = ">=5.3.4,<5.4"
testcontainers = ">=4.4.0,<5"
thefuzz = ">=0.22.1,<0.23"
tiktoken = ">=0.6.0,<0.7"
transformers = ">=4.39.2,<4.40"
uvicorn = ">=0.27.0.post1,<0.27.1"
xmlschema = ">=3.2.0,<3.3"
xmltodict = ">=0.13.0,<0.14"
python-xxhash = ">=3.4.1,<3.5"
pyarrow = "<16" # pandas incompat
requests = "<2.32" # https://github.com/docker/docker-py/issues/3256
pytorch = "<2.3"
pytest-lazy-fixture = ">=0.6.3,<0.7"
loguru = ">=0.7.2,<0.8"
python-docx = ">=1.1.2,<1.2"
markdown = ">=3.6,<4"
pytest-xdist = ">=3.6.1,<3.7"
waitress = ">=3.0.0,<3.1"
pyright = ">=1.1.368,<1.2"

Issue description

Running a large new environment for the first time, on macOS:

❯ pixi run python
  ⠂ default:linux-aarch6400:00:03] resolving conda
  ⠂ default:linux-64     [00:00:03] loading repodata
    ⠙ validate cache       [00:00:00] [━━━━━━━━━━━━━━──────] tesseract (+167)
    ⠙ download & extract   [00:00:00] [────────────────────]      0 B @ 0 B/s    scikit-learn (+100)
    ⠙ installing packages  [00:00:00] [────────────────────]     0/536 sentencepiece-python (+161)                                               × failed to instantiate a prefix for 'default'
  ├─▶ failed to link xmltodict-0.13.0-pyhd8ed1ab_0.tar.bz2
  ├─▶ failed to read 'index.json'
  ╰─▶ Too many open files (os error 24)

❯ wc -l pixi.toml
     105 pixi.toml
❯ wc -l pixi.lock
   25153 pixi.lock
❯ grep platform pixi.toml
platforms = ["linux-64", "linux-aarch64", "osx-arm64"]

Expected behavior

Should do fewer things in parallel so as to not go over macOS ulimit -n

[wolfv]

While I agree that this can be a problem other package managers also frequently run into it (e.g. Yarn).

We could try to figure out the current ulimit though and try to stay below it. Might be a bit of refactor though in rattler ...

[jonashaag]

Maybe an easier workaround would be to improve the error message. Maybe you can globally hardcode something like

if error message contains "Too many open files" or errno = 24
  print "Trying running ulimit -n 12345"

[jonashaag]

I'm not getting this regularly even with small environments

[tdejager]

Do you mean "are getting this regularly"?

[jonashaag]

I meant

I'm now getting this regularly even with small environments

[baszalmstra]

Mm, we do have some code in place to ensure we don't perform too many IO requests. Do you always get the error failed to link .., or is it sometimes something else? Also, what is your current ulimit?

[jonashaag]

macOS default: 256

I was unable to reproduce just now. I'll post the error messages when it happens again

[jonashaag]

Got it again:

❯ mv .pixi/ pixi
❯ pixi run ipython
  ⠁ validate cache       [00:00:00] [━╾──────────────────] tesseract (+521)
  ⠁ download & extract   [00:00:00] [────────────────────]      0 B @ 0 B/s    libhwloc (+10)
  ⠁ installing packages  [00:00:00] [────────────────────]     0/569 libopenvino-auto-batch-plugin (+28)                                         × failed to link pkgconfig-1.5.5-pyhd8ed1ab_4.tar.bz2
  ├─▶ failed to read 'link.json'
  ╰─▶ Too many open files (os error 24)
❯ ulimit -n
256

[wolfv]

I have the same ulimit, but don't see this error often. Your environment does look pretty big though (569 packages).

I wonder if we could defend against this and react on the specific "too many open files" error by waiting, and retrying a few milliseconds later.

[cdeil]

I also get this error with 0.27.0 on MacOS with default

$ ulimit -n
256

Followed this guide: https://holoviz-dev.github.io/panel/developer_guide/index.html

And get this error:

(base) panel $ pixi run test-unit
✨ Pixi task (test-unit in test-312): pytest panel/tests -n logical --dist loadgroup
  ⠁ validate cache       [00:00:00] [━╾──────────────────] pillow (+298)
  ⠁ download & extract   [00:00:00] [────────────────────]      0 B @ 0 B/s    qt6-main (+133)
  ⠁ installing packages  [00:00:00] [────────────────────]     0/452 brotli (+4)                                                                         × failed to link brotli-bin-1.1.0-hb547adb_1.conda
  ├─▶ failed to read 'paths.json'
  ╰─▶ Too many open files (os error 24)

Would be great if you could improve pixi to make this work with defaults on MacOS.

Anyone knows what the workaround is? With Google I find various commands to change system settings for older versions of MacOS but usually with comments that they don't work reliably or are not recommended.

[jonashaag]

ulimit -n 10000

[wolfv]

There is an open PR by @baszalmstra - I need to test it though.

[wolfv]

Bas' PR works for me on macOS so I think we can merge and it'll be in the next release :)

[matthewfeickert]

@wolfv @baszalmstra I'm still seeing this on Apple silicon macOS on v0.28.2 with pixi global unless I manually expand with ulimit -n

~ % pixi self-update
✔ pixi is already up-to-date (version 0.28.2)
~ % pixi global upgrade-all
  × failed to parse '/Users/feickert/.pixi/envs/uv/conda-meta/uv-0.4.2-hd3a8144_0.json'
  ╰─▶ Too many open files (os error 24)

~ % ulimit -n
256
~ % ulimit -n 2048
~ % pixi global upgrade-all
Nothing to upgrade

This is happening with a decently large pixi global collection

pixi global list:

~ % pixi global list
Global install location: /Users/feickert/.pixi
├── anaconda-client 1.12.3
|   └─ exec: conda-server, anaconda, binstar
├── asciinema 2.4.0
|   └─ exec: asciinema
├── bat 0.24.0
|   └─ exec: bat
├── cmake 3.30.3
|   └─ exec: cmake, ctest, cpack, ccmake
├── cookiecutter 2.6.0
|   └─ exec: cookiecutter
├── crane 0.20.2
|   └─ exec: crane
├── curl 8.9.1
|   └─ exec: curl
├── git 2.46.0
|   └─ exec: scalar, git-upload-pack, git-shell, git, git-cvsserver, git-credential-osxkeychain, git-upload-archive, gitk, git-receive-pack
├── grayskull 2.7.1
|   └─ exec: conda-grayskull, conda-greyskull, grayskull, greyskull
├── hepconvert 1.3.7
|   └─ exec: hepconvert
├── hyperfine 1.18.0
|   └─ exec: hyperfine
├── jupyter-repo2docker 2024.7.0
|   └─ exec: repo2docker, jupyter-repo2docker
├── lhapdf 6.5.4
|   └─ exec: lhapdf, lhapdf-config
├── miller 6.12.0
|   └─ exec: mlr
├── ninja 1.12.1
|   └─ exec: ninja
├── pandoc 3.3
|   └─ exec: pandoc-server, pandoc, pandoc-lua
├── pipx 1.7.1
|   └─ exec: pipx
├── pre-commit 3.8.0
|   └─ exec: pre-commit
├── pytube 15.0.0
|   └─ exec: pytube
├── rattler-build 0.20.0
|   └─ exec: rattler-build
├── ruff 0.6.2
|   └─ exec: ruff
├── shellcheck 0.10.0
|   └─ exec: shellcheck
├── tree 2.1.3
|   └─ exec: tree
├── uproot-browser 0.6.7
|   └─ exec: uproot-browser
├── uv 0.4.2
|   └─ exec: uvx, uv
├── visidata 2.11
|   └─ exec: vd, visidata
└── xrootd 5.7.0
    └─ exec: frm_admin, frm_xfragent, xrdcopy, xrdmapc, xrdcrc32c, xrdgsiproxy, xrdpinls, xrdpwdadmin, xrdreplay, wait41, xrootd, xrdcks, xrdcp, xrdpfc_print, xrdfs, mpxstats, xrdacctest, xrdgsitest, xrdsssadmin, xrootd-config, cmsd, frm_xfrd, frm_purged, cconfig, xrdadler32

I'm not sure how much of this is a shortcoming of macOS defaults vs. something that is fixable on the software side.

[baszalmstra]

Do you see anything related to r limit in the logs when you run it with -v ?

If not would you be able to open a new issue? This may be related to the upgrade-all subcommand.

[matthewfeickert]

@baszalmstra I'm not always able to reproduce it — it seems to come and go. I'll be away from the Apple silicon macOS machine this week but I'll try to get a consistent reproducer and make a new issue if needed.

[baszalmstra]

Much appreciated! I will double check the code in the meantime.

[cdeil]

I get this error with pixi 0.28.2 now:

(base) panel $ pixi add pixi-pycharm -v
 INFO pixi::environment: verifying prefix location is unchanged, with prefix file: /Users/cdeil/code/oss/panel/.pixi/envs/default/conda-meta/pixi_env_prefix
 INFO pixi::lock_file::outdated: the dependencies of environment 'default' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'default' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'default' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'default' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-310' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-310' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-310' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-310' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-311' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-311' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-311' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-311' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-312' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-312' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-312' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-312' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-ui' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-ui' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-ui' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-ui' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-core' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-core' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-core' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-core' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-type' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-type' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-type' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'test-type' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'docs' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'docs' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'docs' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'docs' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'build' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'build' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'build' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'build' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'lint' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'lint' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'lint' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'lint' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'lite' for platform linux-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'lite' for platform osx-arm64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'lite' for platform osx-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::outdated: the dependencies of environment 'lite' for platform win-64 are out of date because the requirement 'pixi-pycharm *' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::resolve::uv_resolution_context: uv keyring provider is disabled
 WARN rattler_repodata_gateway::fetch: could not compute BLAKE2 hash of repodata.json file: Too many open files (os error 24). Ignoring cached files...
 WARN rattler_repodata_gateway::fetch: could not compute BLAKE2 hash of repodata.json file: Too many open files (os error 24). Ignoring cached files...
 WARN rattler_repodata_gateway::fetch: could not compute BLAKE2 hash of repodata.json file: Too many open files (os error 24). Ignoring cached files...
  ⠁ test-ui:osx-arm64    [00:00:00] loading repodata
  ⠁ test-ui:linux-64     [00:00:00] [━━━━━━━━━━━━━━━━━━━━]      0 B @ 0 B/s    downloading repodata
  ⠁ lite:osx-arm64       [00:00:00] [━━━━━━━━━━━━━━━━━━━━]      0 B @ 0 B/s    downloading repodata
  ⠁ test-ui:osx-64       [00:00:00] [━━━━━━━━━━━━━━━━━━━━]      0 B @ 0 B/s    downloading repodata
  ⠁ build:osx-arm64      [00:00:00] loading repodata
  ⠁ lite:win-64          [00:00:00] [━━━━━━━━━━━━━━━━━━━━]      0 B @ 0 B/s    downloading repodata
  ⠁ lite:osx-64          [00:00:00] [━━━━━━━━━━━━━━━━━━━━]      0 B @ 0 B/s    downloading repodata
  ⠁ build:win-64         [00:00:00] loading repodata
  ⠁ lite:linux-64        [00:00:00] [━━━━━━━━━━━━━━━━━━━━]      0 B @ 0 B/s    downloading repodata
  ⠁ test-ui:win-64       [00:00:00] [━━━━━━━━━━━━━━━━━━━━]      0 B @ 0 B/s    downloading repodata                                                                                                                   × failed to instantiate a prefix for 'test-ui'
  ├─▶ failed to parse '/Users/cdeil/code/oss/panel/.pixi/envs/test-ui/conda-meta/tinycss2-1.3.0-pyhd8ed1ab_0.json'
  ╰─▶ Too many open files (os error 24)

[matthewfeickert]

If not would you be able to open a new issue? This may be related to the upgrade-all subcommand.

I've opened Issue #2005 for this now. 👍