BUG: some failing builds on windows with micromamba

[h-vetinari]

Since we switched to micromamba, I'm seeing some rare but reproducible failures for particularly heavy builds.

They look like

WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\Library\bin\libcrypto-3-x64.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\Library\bin\libcurl.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\Library\bin\libssh2.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\Library\bin\libssl-3-x64.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\msvcp140.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\vcruntime140.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\vcruntime140_1.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\zlib.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
Packaging libgoogle-cloud-trace-devel
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\Library\bin\libcrypto-3-x64.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\Library\bin\libcurl.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\Library\bin\libssh2.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\Library\bin\libssl-3-x64.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
WARNING:conda.gateways.disk.delete:Could not remove or rename %PREFIX%\msvcp140.dll.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.conda_trash.  Please remove this file manually (you may need to reboot to free file handles)
ERROR:conda.core.link:An error occurred while installing package 'conda-forge::vc14_runtime-14.40.33810-hcc2c482_22'.

seen here:

• Rebuild for libgrpc 1.67 & libprotobuf 5.28.2 google-cloud-cpp-core-feedstock#26
• TEST: arrow v18.0.0rc0 arrow-cpp-feedstock#1432

The problem is 100% reproducible (in arrow), and after undoing the micromamba-changes on arrow, the build passed again (see here).

CC @jaimergp @jjerphan @JohanMabille

[jaimergp]

Also:

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
ERROR:conda.core.link:An error occurred while installing package 'conda-forge::vc14_runtime-14.40.33810-hcc2c482_22'.
Rolling back transaction: ...working... done
Traceback (most recent call last):
  [...]
conda.CondaMultiError: [Errno 13] Permission denied: 'C:\\bld\\apache-arrow_1729721752938\\_h_env\\vcruntime140.dll'

I wonder if there's a lingering open handle somewhere 🤔

[h-vetinari]

I think that's a red herring honestly, or rather just a symptom of a previous problem. And the fact that the exact same recipe builds if reverting the infra changes is an argument in favour of that IMO.

[jaimergp]

Yea, I think the same infra change is behind the two things. I had to add a bit of a hack to reuse the cache in this line, so maybe that's messing with the permissions somehow.

[jaimergp]

Very interesting findings in conda-forge/arrow-cpp-feedstock#1432. If I configure the Windows pipeline so micromamba and conda-build don't use the same cache, it works. This is not ideal because conda will have to redownload some files for the conda-build envs.

If I make them share the cache in some way (either by moving the micromamba cache over to conda's location as currently done in conda-smithy, or by configuring micromamba to use conda's location from the beginning), we hit the error in this issue.

It looks like micromamba is linking things in a way that prevents conda / conda-build from reusing those cached files.