"No module named 'packaging.metadata'" when running 1.8.3 on Mac, Linux, and Windows #101
Comments
|
@l0b0, I appreciate you reporting this. Yours seems like a complicated case! Looking at your CI, it seems like you first create a conda environment, and then install But then you install a bunch of packages from your poetry lock file: This results in mixing a bunch of packages from conda-forge with a bunch from PyPI, which is typically not something that works well. The result is some downgrades and presumably a bunch of reinstalls: We would probably need to see the output of poetry-feedstock/recipe/meta.yaml Line 46 in 95bae92 so if this had been a pure conda environment, we would not have let this downgrade occur. But we can't control what PyPI packages you install into a conda environment. It seems like your workflow would be more compatible with starting with a very simple conda environment (e.g. just with python and pip) and installing all of your dependencies from PyPI. Importantly, it seems like you should install I'm happy to help where I can but I don't think this is a conda-forge problem (or a poetry problem or a PyPI problem) but an issue of mixing packages from different distributions. |
|
Thank you very much for your thorough response, @xylar! Some background: (tl;dr: If this can't be done easily and reliably with Conda I'll have to
This has resulted in the following decisions:
Some observations have also played into this (some of them may be wrong or outdated; I'd welcome constructive feedback):
Together, these mean that Conda is the only obvious choice to install GDAL. However
The result is that I want to use Conda to install GDAL, but also that I want to use some other solution to install all the other packages. Pip is missing many useful features of modern package managers (locking using hashes by default, separating top-level and transitive dependencies, and separating development and production dependencies are just the biggest gripes), so that's out. I know Poetry reasonably well, and there's poetry2nix to avoid the need to reconfigure anything to install Python packages using Nix. |
|
@l0b0, do you think you would have more luck if you install only GDAL from conda but I use conda-forge almost exclusively so I don't have a lot of personal experience on when mixing conda-forge and PyPI packages breaks and when it works, but we get a lot of complaints from users who try to mix packages between the two. I think if you do continue to mix packages from both, you're going to have to take responsibility for debugging things that go wrong because I don't think either the folks that release their packages on PyPI nor conda-forge maintainers are interested in figuring out how to make a hybrid work. I think the trouble you are running into with I don't know anything about Nix. You said that conda environments too often don't work reliably. That can certainly happen as packages get marked as broken. I think it's substantially rarer than it used to be but I can't say from experience how that might compare with PyPI. GDAL has an especially complicated and fragile dependency chain so that may be a package that, together with other dependencies, could become challenging. You said that too many of your dependencies aren't available on conda-forge. I would be willing to help get them built on conda-forge if you think that could become a feasible path forward. It would be important to know how many packages we're talking about and how many are pure python vs. something more complicated. |
As [recommended by the documentation](https://python-poetry.org/docs/#ci-recommendations> and [@xylar](conda-forge/poetry-feedstock#101 (comment)).
As [recommended by the documentation](https://python-poetry.org/docs/#ci-recommendations) and [@xylar](conda-forge/poetry-feedstock#101 (comment)).
As [recommended by the documentation](https://python-poetry.org/docs/#ci-recommendations) and [@xylar](conda-forge/poetry-feedstock#101 (comment)). This should avoid the "No such file or directory: '/usr/local/miniconda/envs/test/lib/python3.[…]/site-packages/[…].dist-info'" errors which probably comes from the interaction between Conda and Poetry.
|
Thank you very much for your help, @xylar!
It's possible. I'm pretty sure I've only seen this issue when using Poetry with Conda, though.
That would be everything in the two ❯ grep --no-filename --only-matching --perl-regexp '(?<=^name = ")[^"]+' flooding/sentinel*_water_extraction/poetry.lock | sort -u
affine
annotated-types
anyio
appnope
argon2-cffi
argon2-cffi-bindings
arrow
asttokens
async-lru
attrs
babel
backcall
beautifulsoup4
bleach
cachetools
certifi
cffi
charset-normalizer
click
click-plugins
cligj
colorama
color-operations
comm
contourpy
cycler
debugpy
decorator
defusedxml
entrypoints
executing
fastjsonschema
fiona
fonttools
fqdn
gdal
geopandas
h11
httpcore
httpx
idna
importlib-metadata
importlib-resources
ipykernel
ipython
ipython-genutils
ipywidgets
isoduration
jedi
jinja2
json5
jsonpointer
jsonschema
jsonschema-specifications
jupyter
jupyter-client
jupyter-console
jupyter-core
jupyter-events
jupyterlab
jupyterlab-pygments
jupyterlab-server
jupyterlab-widgets
jupyter-lsp
jupyter-server
jupyter-server-terminals
kiwisolver
markupsafe
matplotlib
matplotlib-inline
mistune
morecantile
munch
nbclassic
nbclient
nbconvert
nbformat
nest-asyncio
notebook
notebook-shim
numexpr
numpy
overrides
packaging
pandas
pandocfilters
parso
pexpect
pickleshare
pillow
platformdirs
prometheus-client
prompt-toolkit
psutil
ptyprocess
pure-eval
pycparser
pydantic
pydantic-core
pygments
pyparsing
pyproj
pyrsistent
pystac
python-dateutil
python-json-logger
pytz
pywin32
pywinpty
pyyaml
pyzmq
qtconsole
qtpy
rasterio
referencing
requests
rfc3339-validator
rfc3986
rfc3986-validator
rio-tiler
rpds-py
sat-search
sat-stac
send2trash
setuptools
shapely
six
sniffio
snuggs
soupsieve
stack-data
terminado
tinycss2
tomli
tornado
traitlets
typing-extensions
uri-template
urllib3
wcwidth
webcolors
webencodings
websocket-client
widgetsnbextension
zipp |
|
Oof, that’s a lot! |
|
Not really, it's one of the smallest dependency Python chains I've dealt with. These are the top-level packages: ❯ grep -F --no-filename '= "*"' flooding/sentinel*_water_extraction/pyproject.toml | cut -d ' ' -f1 | sort -u
gdal
geopandas
jupyter
jupyterlab
matplotlib
rio-tiler
sat-search
shapelyBasically, if these could all be installed with Conda, including all their dependencies, that would add up to the 200+ packages in my previous comment. |
|
Oh, much more manageabe! I’ll see what I can do. |
|
I’m guessing these are missing? rio-tiler |
I haven't looked into what's missing; I always assumed it was going to be an unsustainable amount of work. Also, please don't feel obliged to work on any of this! I don't even know whether a full Conda solution would be preferable to the current one. |
All of these are available on conda-forge: https://conda-forge.org/packages/ |
|
@l0b0 can you see what goes wrong when you try to install everything from conda-forge? I'm happy to debug incompatibilities and try to make your installation from conda-forge more robust. It should be possible to use yaml files to install very specific environments (either with specific versions of packages pinned, down to the build hash or with tight constraints on version ranges), similar to your lock files. I suspect unless you get all packages either from PyPI or from conda-forge that things will never be completely robust. I realize that the nix question remains. |
Sincere thanks for that! I'm sorry to say I don't think I will spend time on this right now, based on issues mentioned below.
It would need at least these features of Poetry:
I think I do?
I could easily live without Nix support if Conda gave some other advantages over the current setup. Another issue is that Dependabot does not support Conda, so future version upgrades would be manual. |
|
@l0b0, I'm glad you have something that works at least. |
As [recommended by the documentation](https://python-poetry.org/docs/#ci-recommendations) and [@xylar](conda-forge/poetry-feedstock#101 (comment)). This should avoid the "No such file or directory: '/usr/local/miniconda/envs/test/lib/python3.[…]/site-packages/[…].dist-info'" errors which probably comes from the interaction between Conda and Poetry.
Solution to issue cannot be found in the documentation.
Issue
After upgrading Poetry from 1.3.2 to 1.8.3, I'm unable to run it. Output from GitHub Actions:
Discussion with the Poetry community.
Workaround: Upgrade to 1.7.1, which seems to be the latest non-broken version.
Installed packages
Environment info
The text was updated successfully, but these errors were encountered: