Pip-installed Poetry encountering EnvironmentError "Permission denied" on Windows when upgrading cffi

[PAStheLoD]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 10

• Poetry version: 1.0.10

Issue

I've just encountered this problem while running poetry update, which upgraded the cffi package. Using pip 20.2.3 on python 3.9b5 on Windows 10, while using poetry 1.0.10.

PS C:\pas\projects\xxxx\ci> .\venv-windows\Scripts\python.exe -m poetry update
Skipping virtualenv creation, as specified in config file.
Updating dependencies
Resolving dependencies...

Writing lock file


Package operations: 0 installs, 5 updates, 0 removals

  - Updating attrs (19.3.0 -> 20.2.0)
  - Updating cffi (1.14.0 -> 1.14.2)

[EnvCommandError]
Command ['C:\\pas\\projects\\xxxx\\ci\\venv-windows\\Scripts\\python.exe', '-m', 'pip', 'install', '--no-deps', '-U', 'cffi==1.14.2'] errored with the following return code 1, and output:
Collecting cffi==1.14.2
  Downloading cffi-1.14.2.tar.gz (470 kB)
Using legacy 'setup.py install' for cffi, since package 'wheel' is not installed.
Installing collected packages: cffi
  Attempting uninstall: cffi
    Found existing installation: cffi 1.14.0
    Uninstalling cffi-1.14.0:
      Successfully uninstalled cffi-1.14.0
    Running setup.py install for cffi: started
    Running setup.py install for cffi: finished with status 'done'
ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Permission denied: 'C:\\Users\\Admin\\AppData\\Local\\Temp\\pip-uninstall-8rjju1cu\\_cffi_backend.cp39-win_amd64.pyd'

Of course it's not the best practice to install poetry via pip. Maybe providing a command line argument to ignore these errors might be worth it?

[abn]

@PAStheLoD not sure that is a poetry problem tbh. Looks like pip trying to install cffi fails.

You should be able to reproduce the issue when running the command from this error manually.

Command ['C:\\pas\\projects\\xxxx\\ci\\venv-windows\\Scripts\\python.exe', '-m', 'pip', 'install', '--no-deps', '-U', 'cffi==1.14.2'] errored with the following return code 1, and output:

Also your experience on poetry from master might be better for this scenario. There are some improvements on how we install packages.

[PAStheLoD]

Sorry for the incomplete/unclear report. What I haven't explicitly stated is that poetry is installed into the same virtualenv that it manipulates. So I guess it is using that particular file. pip itself vendors most/all (?) of its dependencies (if I understand the pip repo correctly).

Thanks for the quick reply! If I'll have some time I'll try to reproduce this with just pip, and will check out master too.

[jeremad]

Exact same issue with poetry, cffi and windows, I try to create a minimum reproduction

[abn]

@PAStheLoD @jeremad as I mentioned before this is to do with pip not being able to install the package from sdist, caused due to the permission error. This is not something poetry can control. Would be great to know the root cause of the permission error.

As for the failures, this might no longer be an issue with the latest pre-release since the installation logic and venv creation (#2666) has changed. Please try with the pre-release.

[PAStheLoD]

The install part itself successfully happens, it's the removal of the temp pip-uninstall-[nonce] thing. Presumably pip moves the potentially still in-use package to a temp folder. Then tries to delete it on exit, and fails, because it's used (by poetry or whatever invoked pip from the same venv).

At least this is my (admittedly incomplete) understanding. I tried to find some relevant discussion about this post-run cleanup mechanism, but only found this pypa/pip#7567 , and it seems this Windows-only thing is still not implemented (and likely it wouldn't be able to handle this case anyway).

So probably the main problem is that poetry should not both manage and run from the same venv. Thanks again!

[abn]

The install part itself successfully happens, it's the removal of the temp pip-uninstall-[nonce] thing. Presumably pip moves the potentially still in-use package to a temp folder. Then tries to delete it on exit, and fails, because it's used (by poetry or whatever invoked pip from the same venv).

Since imported modules are in memory, deleting their files should not matter unless it is deleting an exe or something, which should not be happening anyway.

At least this is my (admittedly incomplete) understanding. I tried to find some relevant discussion about this post-run cleanup mechanism, but only found this pypa/pip#7567 , and it seems this Windows-only thing is still not implemented (and likely it wouldn't be able to handle this case anyway).

So probably the main problem is that poetry should not both manage and run from the same venv. Thanks again!

Poetry in theory should be running outside the venv anyway, unless it was explicitly installed into the venv.

[PAStheLoD]

Oh, I thought .pyd files were similar to .dll files in this regard. (So that they get memory mapped into the process space, but even after reading about it I'm not sure. If they are created with Cython then they are not bytecode but machine code.)

Yes, in my case it was explicitly installed into the same venv.

[Korijn]

I have the same issue (since today), just in a blank new virtualenv, when running poetry run python -m pip install -U pip:

I just freshly installed poetry 1.0.10, python 3.6.8 (64bit), Windows 10

[abn]

@Korijn what happens when you do the following? (My windows is a bit rusty!)

python3.6 -m venv .venv
.venv\Scripts\python.exe -m pip install -U pip

[Korijn]

@Korijn what happens when you do the following? (My windows is a bit rusty!)

python3.6 -m venv .venv
.venv\Scripts\python.exe -m pip install -U pip

Seems to be fine... huh...

[abn]

@Korijn confirming that is python 3.6?

This is bizarre. I'll try reproducing it in a windows VM.

[Toxe]

I have a similar issue. Newest version of poetry and Windows 10 Home (version 20H2), a fresh environment and Python 3.8.6. Using PowerShell in the new Windows Terminal app.

When running poetry update I get:

Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 0 installs, 23 updates, 1 removal

  • Removing more-itertools (8.5.0)
  • Updating colorama (0.4.3 -> 0.4.4)
  • Updating isort (5.5.2 -> 5.6.4)
  • Updating toml (0.10.1 -> 0.10.2)
  • Updating prompt-toolkit (3.0.7 -> 3.0.8)
  • Updating pygments (2.7.1 -> 2.7.2)
  • Updating traitlets (5.0.4 -> 5.0.5)
  • Updating sqlalchemy (1.3.19 -> 1.3.20)
  • Updating attrs (20.2.0 -> 20.3.0)
  • Updating certifi (2020.6.20 -> 2020.11.8)
  • Updating iniconfig (1.0.1 -> 1.1.1)
  • Updating kiwisolver (1.2.0 -> 1.3.1)
  • Updating packaging (20.4 -> 20.7)
  • Updating numpy (1.19.2 -> 1.19.4)
  • Updating pathspec (0.8.0 -> 0.8.1)
  • Updating ipython (7.18.1 -> 7.19.0)
  • Updating regex (2020.7.14 -> 2020.11.13)
  • Updating urllib3 (1.25.10 -> 1.26.2)
  • Updating pillow (7.2.0 -> 8.0.1)

  EnvCommandError

  Command C:\Users\toxe\AppData\Local\pypoetry\Cache\virtualenvs\gps-tracks-rest-api-mrZqQZX0-py3.8\Scripts\pip.exe install --no-deps -U file:///C:/Users/toxe/AppData/Local/pypoetry/Cache/artifacts/6e/95/18/3db0ce3882c0ae273d01c0efa35e8bf56a95390ca600c814ace94cea9a/regex-2020.11.13-cp38-cp38-win_amd64.whl errored with the following return code 1, and output:
  Processing c:\users\toxe\appdata\local\pypoetry\cache\artifacts\6e\95\18\3db0ce3882c0ae273d01c0efa35e8bf56a95390ca600c814ace94cea9a\regex-2020.11.13-cp38-cp38-win_amd64.whl
  Installing collected packages: regex
    Attempting uninstall: regex
      Found existing installation: regex 2020.7.14
      Uninstalling regex-2020.7.14:
        Successfully uninstalled regex-2020.7.14
  ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'C:\\Users\\toxe\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\gps-tracks-rest-api-mrZqQZX0-py3.8\\Lib\\site-packages\\~egex\\_regex.cp38-win_amd64.pyd'
  Consider using the `--user` option or check the permissions.

  WARNING: You are using pip version 20.2.2; however, version 20.3.1 is available.
  You should consider upgrading via the 'C:\Users\toxe\AppData\Local\pypoetry\Cache\virtualenvs\gps-tracks-rest-api-mrZqQZX0-py3.8\Scripts\python.exe -m pip install --upgrade pip' command.


  at ~\.poetry\lib\poetry\utils\env.py:1074 in _run
      1070│                 output = subprocess.check_output(
      1071│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1072│                 )
      1073│         except CalledProcessError as e:
    → 1074│             raise EnvCommandError(e, input=input_)
      1075│
      1076│         return decode(output)
      1077│
      1078│     def execute(self, bin, *args, **kwargs):

If anyone wants to reproduce this:

• clone https://github.com/Toxe/gps-tracks.git
• checkout commit 991a58e
• go into the rest-api folder
• create your environment and perform a poetry update from there

[sinoroc]

@Toxe Looks like a pip issue somehow, so maybe out of poetry's control. Looks like at the moment the error occurs, poetry had launched this command in a subprocess:
C:\Users\toxe\AppData\Local\pypoetry\Cache\virtualenvs\gps-tracks-rest-api-mrZqQZX0-py3.8\Scripts\pip.exe install --no-deps -U file:///C:/Users/toxe/AppData/Local/pypoetry/Cache/artifacts/6e/95/18/3db0ce3882c0ae273d01c0efa35e8bf56a95390ca600c814ace94cea9a/regex-2020.11.13-cp38-cp38-win_amd64.whl
and then pip fails while trying to delete this file for whatever reason:
'C:\\Users\\toxe\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\gps-tracks-rest-api-mrZqQZX0-py3.8\\Lib\\site-packages\\~egex\\_regex.cp38-win_amd64.pyd.

Have you tried deleting that file? Are you sure the issue is reproducible in a fresh environment?

[Toxe]

@sinoroc

Are you sure the issue is reproducible in a fresh environment?

Yes, 100% sure. Actually the above log was from a fresh environment.

[sinoroc]

Maybe the _regex.cp38-win_amd64.pyd is still being accessed somehow. Maybe some clues on pip's ticket tracker...

• pip fails on windows due to interaction with pyinvoke on windows pypa/pip#8878

Does poetry show --tree (before running the update) show what depends on regex? Can't test it myself, I don't have Windows.

[Toxe]

This is the peotry show --tree part of the output where regex is listed:

black 20.8b1 The uncompromising code formatter.
|-- appdirs *
|-- click >=7.1.2
|-- mypy-extensions >=0.4.3
|-- pathspec >=0.6,<1
|-- regex >=2020.1.8
|-- toml >=0.10.1
|-- typed-ast >=1.4.0
`-- typing-extensions >=3.7.4

[sinoroc]

@Toxe OK, nothing all too meaningful there. Does not give any more clues.

Seems to me like the only sure thing, is that there is a file that can not be deleted when pip is trying to uninstall something as part of an upgrade. And to me it is not sure that the issue lies on poetry's side.

[Toxe]

I agree, at this point I suspect some weird Windows access permissions issue.

[msachsenhauser]

FYI: I've encountered this problem many times over various conda/python/pip/poetry -versions and it's still a thing! Is there a known issue somewhere?
@Toxe Some weird permissions issue is plausible.

Latest setup with this error:

• Windows 10H2
• conda 4.10.3
• python 3.9.13
• poetry 1.1.14
• pip 22.2.2

Solution:

• manually delete folder "<conda-venv>/Lib/site-packages/~egex"

[neersighted]

I'm going to close this one as finding a reproducer has been inconclusive, and more importantly, most of the people commenting are likely experiencing #1031. I'm dubious that this will ever be reproducible, as the only user who has provided details is installing Poetry into the same environment, which is obviously fraught as an overlapping dependency tree will cause Poetry to pull the rug out from under itself.

[SergeyHein]

I'm experiencing exactly the same problem on
Window 10
Python 3.8
Poetry (version 1.3.2)

Updating dependencies
Resolving dependencies... (30.4s)

Writing lock file

Package operations: 26 installs, 2 updates, 0 removals

  • Updating charset-normalizer (3.0.1 -> 2.0.12): Failed

  CalledProcessError

  Command 'C:\XXX_VENV\Scripts\python.exe -m pip install --disable-pip-version-check --isolated --no-input --prefix C:\XXX_VENV --upgrade --no-deps %LOCALAPPDATA%\pypoetry\Cache\artifacts\56\d7\9b\d9f1495af5a4b39024defc8404085361a30368aaa800fecb9fc228c5e0\charset_normalizer-2.0.12-py3-none-any.whl' returned non-zero exit status 1.

  at C:\python38\lib\subprocess.py:516 in run
       512│             # We don't call process.wait() as .__exit__ does that for us.
       513│             raise
       514│         retcode = process.poll()
       515│         if check and retcode:
    →  516│             raise CalledProcessError(retcode, process.args,
       517│                                      output=stdout, stderr=stderr)
       518│     return CompletedProcess(process.args, retcode, stdout, stderr)
       519│
       520│

The following error occurred when trying to handle this error:


  EnvCommandError

  Command C:\XXX_VENV\Scripts\python.exe -m pip install --disable-pip-version-check --isolated --no-input --prefix C:\XXX_VENV --upgrade --no-deps %LOCALAPPDATA%\pypoetry\Cache\artifacts\56\d7\9b\d9f1495af5a4b39024defc8404085361a30368aaa800fecb9fc228c5e0\charset_normalizer-2.0.12-py3-none-any.whl errored with the following return code 1, and output:
  Looking in indexes: http://pypi.ad.solactive.com/packages/stable
  Processing %LOCALAPPDATA%\pypoetry\cache\artifacts\56\d7\9b\d9f1495af5a4b39024defc8404085361a30368aaa800fecb9fc228c5e0\charset_normalizer-2.0.12-py3-none-any.whl
  Installing collected packages: charset-normalizer
    Attempting uninstall: charset-normalizer
      Found existing installation: charset-normalizer 3.0.1
      Uninstalling charset-normalizer-3.0.1:
        Successfully uninstalled charset-normalizer-3.0.1
  ERROR: Could not install packages due to an OSError: [WinError 5] Access is denied: 'C:\\XXX_VENV\\Lib\\site-packages\\~harset_normalizer\\md.cp38-win_amd64.pyd'
  Check the permissions.



  at ~\\...\\lib\site-packages\poetry\utils\env.py:1540 in _run
      1536│                 output = subprocess.check_output(
      1537│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1538│                 )
      1539│         except CalledProcessError as e:
    → 1540│             raise EnvCommandError(e, input=input_)
      1541│
      1542│         return decode(output)
      1543│
      1544│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:


  PoetryException

  Failed to install %LOCALAPPDATA%/pypoetry/Cache/artifacts/56/d7/9b/d9f1495af5a4b39024defc8404085361a30368aaa800fecb9fc228c5e0/charset_normalizer-2.0.12-py3-none-any.whl

  at ~\\...\\lib\site-packages\poetry\utils\pip.py:58 in pip_install
       54│
       55│     try:
       56│         return environment.run_pip(*args)
       57│     except EnvCommandError as e:
    →  58│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       59│

  • Installing pycparser (2.21)

Please let me know if you need more information to re-open the ticket.
Thank you.

[mohghaderi]

Hi @SergeyHein, I am facing the same issue on my build server.
If you are on local machine, you can delete charset-normalizer safely, and it will work.
I am not sure what I can do on the build server.

[LordFckHelmchen]

@mohghaderi - I have the exact same problem. We "fixed" it by repeatedly trying the install call. Most of the time this works as you'll fail on different packages.

There is also a related SO question: https://stackoverflow.com/questions/64151403/poetry-update-has-access-denied
Unfortunately without a good answer...

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.