connection error while installing environment

[tmirko]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

pyproject.toml.txt

~/git-projects/sample (feat/conda2pixi)$ pixi --version
pixi 0.34.0
~/git-projects/sample (feat/conda2pixi)$ pixi install -e prod
  ✔ validate cache       177 packages in 53s 805ms
  ⠉ download & extract   [00:00:55] [━╾──────────────────] 48.42 MiB @ 1.03 MiB/s pyspark (+281)
  ⠉ installing packages  [00:00:55] [━━━━━━━━────────────]   177/459                                                            × failed to fetch annotated-types-0.7.0-pyhd8ed1ab_0.conda
  ├─▶ error sending request for url (https://conda.anaconda.org/conda-forge/noarch/annotated-types-0.7.0-pyhd8ed1ab_0.conda)
  ├─▶ client error (SendRequest)
  ├─▶ connection error
  ╰─▶ peer closed connection without sending TLS close_notify: https://docs.rs/rustls/latest/rustls/manual/_03_howto/
      index.html#unexpected-eof

Issue description

connection error after executing pixi install command, can't be replicated exactly, sometimes fails on a different conda-forge dependency.

Expected behavior

Successful environment build

[ruben-arts]

Hey @tmirko, how did you install pixi? And are there any special rules on your network?

You can always try with --tls-no-verify to see if that works.

[tmirko]

Hi @ruben-arts , I've tested it as an installation in WSL ubuntu image as well as starting from https://ghcr.io/prefix-dev/pixi:0.33.0-bookworm docker image. I'm using company's proxy

--tls-no-verify doesn't help, same error.

[ruben-arts]

Could you try https://prefix.dev/conda-forge as a channel?

[benmoss]

I also get this periodically. Does retrying cause it to succeed, or at least get to another package before failing?

[milicevica23]

Hi
@ruben-arts answer on your question is it works with the conda-forge default chanell

@benmoss we also see it just periodically on my machine but on @tmirko it was the whole time

We spent today a bit smoke testing and get to some potential scenario where we think that the problem is:

private artifactory (with some network security) + slow internet is creating the problem that either some network packages get lost or artifactory stops the connectivity to the client which has open multiple connections to it

I don't know if this makes sense at all but by @tmirko connecting to mobile phone (faster internet) solved the problem for now and he was able to install the environment

Our questions are:

• Do you see another way to debug/test it with some extra data which we can get out of pixi or underalying rust process?
• Do we have a posabillity to slow down or setup the max connections to the artifactory in order to back the pressure?

[benmoss]

Could this be related to denoland/deno#6197 ? Or at least the underlying issue there with rustls?

[wolfv]

I think @pavelzw and friends also had a number of issues with bad internet / private artifactory / lots of connections at the same time.

I think what we can do is:

• add config parameters to control concurrency when downloading better (https://github.com/wolfv/rattler/blob/cf0e2f0ef36eb34be9703ebbb95fe57560243232/crates/rattler/src/install/installer/mod.rs#L96)
• try to decouple downloading from extracting more (that might fix another issue that we've seen where donwloading becomes slow because extraction takes very long)

We'll prioritize these items soon.

[pavelzw]

i've also had problems with the exact same setup (artifactory + slowish internet)

i debugged some stuff as well with pixi install -vv and got some indication why the issue occurs.

2024-10-17T09:41:27.9270669Z WARN rattler_cache::package_cache: failed to download and extract https://conda.anaconda.org/conda-forge/linux-64/pynng-0.8.0-py312h2ec8cdc_4.conda to /home/runner/.cache/rattler/cache/pkgs/pynng-0.8.0-py312h2ec8cdc_4: an io error occurred: error decoding response body. Retry #1, Sleeping 543.56173ms until the next attempt...
2024-10-17T09:41:27.9350727Z WARN rattler_cache::package_cache: failed to download and extract https://conda.anaconda.org/conda-forge/linux-64/ocl-icd-2.3.2-hd590300_1.conda to /home/runner/.cache/rattler/cache/pkgs/ocl-icd-2.3.2-hd590300_1: an io error occurred: failed to iterate over archive. Retry #1, Sleeping 102.437491ms until the next attempt...
2024-10-17T09:41:27.9357826Z WARN rattler_cache::package_cache: failed to download and extract https://conda.anaconda.org/conda-forge/linux-64/scikit-learn-1.5.2-py312h7a48858_1.conda to /home/runner/.cache/rattler/cache/pkgs/scikit-learn-1.5.2-py312h7a48858_1: an io error occurred: failed to unpack `/home/runner/.cache/rattler/cache/pkgs/scikit-learn-1.5.2-py312h7a48858_1/lib/python3.12/site-packages/sklearn/ensemble/_hist_gradient_boosting/_predictor.cpython-312-x86_64-linux-gnu.so`. Retry #1, Sleeping 340.11399ms until the next attempt...
2024-10-17T09:41:27.9365965Z WARN rattler_cache::package_cache: failed to download and extract https://conda.anaconda.org/conda-forge/linux-64/numba-0.60.0-py312h83e6fd3_0.conda to /home/runner/.cache/rattler/cache/pkgs/numba-0.60.0-py312h83e6fd3_0: an io error occurred: failed to unpack `/home/runner/.cache/rattler/cache/pkgs/numba-0.60.0-py312h83e6fd3_0/lib/python3.12/site-packages/numba/tests/__pycache__/test_unicode_array.cpython-312.pyc`. Retry #1, Sleeping 983.529055ms until the next attempt...

see https://discord.com/channels/1082332781146800168/1296522386601939055/1296522470559580242 or setup-pixi-error.txt
for the full logs

my assumption would be that the slow internet leads to corrupt packages that pixi can't extract, thus an IO error

xref #1994

[pavelzw]

what i found interesting in my findings was that the logs only say Retry #1 in ... so i assume after the first retry it works again.
the package where the error then occurs (in my example libssh2) was not retried before, why is this the case?