Support for mutually exclusive dependencies across groups (e.g. path dependencies during development only)

[hozn]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Question

I believe this issue is related to #668 -- or some of the comments therein -- but I believe my workflow / use case is also a little different from the origin of that issue.

Here's the background / what I'm trying to accomplish:

• I have a large modular python project. It is generally a main application with dependencies on other components that we maintain -- shared libraries, essentially.
• Adding a new feature in the main application often involves changing one or more of these shared libraries.
• During development, I'd like to be able to experiment with different code in those shared libraries and integration test it with my main application before actually writing tests for / committing / tagging / pushing / etc. the changes to those shared libraries.

Here's how I would do this with setuptools / pip:

• Ensure that my main application (my-app e.g.) has a requirements.txt file that has loose enough versions of my shared libraries so that it'll be satisfied by the next version that I'm working on. For example, if my-app will use an updated my-lib1 version 4.1 and my-lib2 version 1.12.3, I might have the following inmy-app/requirements.txt:

  my-lib1 ^4.1
  my-lib2 ^1.12.3
  

• Setup a virtualenv for my top-level application and activate it.
• Change into each shared library directory and run pip install -r requirements.txt && python setup.py develop (Sometimes pip install -e . would also work, but the behavior was a little more consistent actually using python setup.py develop. Argh, the mess that is python packaging !?!)
• Finally I'd run pip install -r requirements.txt in my main application; because I'd already done python setup.py develop I'd have installed (symlinked) in the right versions to satisfy for my-lib1 and my-lib2.
• After finishing development, I'd package up the shared libraries and deploy them to our registry; then when my-app was pushed to build server, it'd find the versions it needed.

This all works reasonably well, though the initial environment setup is a pain. I'm really hoping to move to Poetry but I also feel that I "need" an equivalent to the above.

I can use "path dependencies" to point to my local shared libraries, but then these aren't going to work when it actually goes to build on the CI server (or on anyone else's workstation if they organize their files differently):

[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = { path = "../lib1/", develop = true }
my-lib2 = { path = "../lib2/", develop = true }
# THIS ONLY WORKS ON MY WORKSTATION

If I specify the same dependency in [tool.poetry.dependencies] and [tool.poetry.dev-dependencies], it seems to always pick the one from the non-dev deps:

[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = "^4.1"
my-lib2 = "^1.12.3"

[tool.poetry.dev-dependencies]
my-lib1 = { path = "../lib1/", develop = true }
my-lib2 = { path = "../lib2/", develop = true }

# IT ALWAYS TRIES TO FIND MATCHING VERSIONS ON THE PYPI SERVER, DOESN'T USE THE PATHS

But even if this did work, ideally the solution wouldn't involve me using path dependencies at all, since (in this case) these are specific to how my workstation is setup -- a coworker may organize things differently.

So, is there a current solution to solving this problem? Perhaps I've missed something in reading the docs. Thanks for any help!

[ankostis]

The solution to this existing problem requires some kind tof "redirection": the "path dependencies" contain the exact path, while another "type of dependency" would allow each user to configure ("redirect to") its exact path.

Actually, not even a new type of dependency is needed, as long as the user can override and re-direct any regular dependency to its own path.

[hozn]

@ankostis , I agree that allowing the user to override -- e.g. with commandline arguments -- during the install step would meet my requirements here. Something like:

poetry install --path-dep ../path/to/my-lib1 --path-dep ../path/to/my-lib2

It would probably still be required that the dependency at that local path matched the version specified in my pyproject.toml file.

[avanzal]

I'm currently facing the same challenges as described.

Basically, I have a number of library packages and a number of app packages using those libraries, and would like to:

1. Use poetry to install dependencies during local development (where app and library packages are on disk)
2. Use poetry to build dists for the library packages

The problem seems to be a disconnect between path dependencies, poetry install, and poetry build:

1. While developing, it's pretty handy to use path dependencies (instant updates without build+publish, edits across multiple libraries at once, etc); but
2. If I specify any path dependencies in pyproject, I can't do the required poetry build (doesn't allow path dependencies)
3. If I specify only non-path dependencies in pyproject, poetry install / poetry update will fetch from the configured repository instead of local dirs, which is counter to point 1

I've tried (and failed) on some workarounds:

• I can't use poetry add --path, because the dependency already specified in pyproject. Also it's pretty painful to do this for each library.
• I tried adding the libraries as non-path dependencies to tools.poetry.dependencies and as path dependencies to tools.poetry.dev-dependencies, hoping poetry build and poetry install would pick from these separately, but this doesn't work (poetry build still fails)

Found some partial / potential workarounds, though none of these cover all requirements:

• Use poetry run pip install -e ... for local dependencies, and write some scripting automate this somewhat as it would be pretty painful to do by hand for all the connected libraries.
• Use only non-path dependencies in the library pyprojects, and add all local libraries (including sub-dependencies) as path dependencies to the app's pyproject. Running poetry install on the app uses local disk via the specified path dependencies. However, in my use case each library package has its own test suite, so they also need to do a poetry install, and since they're now non-path dependencies this doesn't work with local development.
• Each developer could run their own local PyPi server and publish their dev packages to it. Really not ideal though, especially if you're changing the libraries a lot.

And also:

• It would be really useful if path dependencies also supported version = "..." (eg. xyz-log = {path="../xyz-log", version="^1.0.0"}), would help with branched code etc.

I think this problem would be solved if there was a configurable package resolver - eg. you could configure it to first look up the package name in a particular directory and if it exists then install it as path dependency, else go look for a git repo <base-url>/<package-name>.git, else go look in PyPi, etc - whatever the user configures. This is on similar lines to the command line parameter idea of poetry install --path-dep ... from the comments

[bersace]

I'm facing this issue while maintaining two poetry-managed libraries. I did the following:

~/src/dep$ poetry install
~/src/dep$ cd ../main/
~/src/main$ poetry install
~/src/main$ echo $HOME/src/dep >> $VIRTUAL_ENV/lib/python3.6/site-packages/easy-install.pth
~/src/main$ python -c 'import dep; print(dep.__file__)'
/home/.../src/deps/dep.py
~/src/main$

That's not the best solution, but still fire and forget.

[pbryan]

I too am in a similar situation. I'm substituting a dependency (e.g. x = "^1.3") with a path dependency (x = { path = "../x") in pyproject.toml, which works. Ideally, there would be a method of overriding any dependency (or dev-dependency) in a "poetry-installed" project without modifying pyproject.toml (maybe in some kind of "override" file).

[stale]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[hozn]

Um, yes this is still relevant! Unless something has changed this is still just as broken/incomplete as it was when the issue was created.

[hozn]

@sdispater , could we get feedback on this issue?

[freddrake]

I'm using path dependencies similar to some of the others who've followed up is this issue already, but am interesting in having those dependencies built and installed into the virtual env just like anything else, not as "editable" installs using .egg-link metadata. My goal is to be able to use the virtual env as input to a process that builds a platform package (.deb in my case).

Being able to control the type of installation separately from the type of source would be quite handy.

[niander]

Any update on this?

[idmitrievsky]

Can anyone weigh in on what are the obstacles for this functionality, so we could begin working on eliminating them?

[abn]

@hozn Your use case sounds very similar to one I have porposed at #2270. It would be good to get more eyes on that one and possibly feedback as a way to solve this.

Additionally, there is also #755 which might be another solution once implemented. However, that will only work if you are okay working with VCS copies. This would also imply that you can simply use something like git-submodules, which might not be ideal.

@idmitrievsky at the moment, I do not think there is a specific solution proposed here. If #2270 would be a solution that would be awesome, it is something I do want to work on soon once we have some feedback.

If poetry were to allow for dev dependencies to override specified dependencies, this would sort of break the locking mechanism itself. A possible compromise might be to allow for a local configuration that allows for the pre-pending of certain paths in the project's virtualenv without using it to determine dependencies or update lockfile etc. If that is something that is desirable, I do think that will sit well inside a plugin once the plugin system is completed.

[idmitrievsky]

@abn thank you for your response! I will keep an eye on the plugin system then.

[applied-mathematician]

Add me to the list of those facing this issue. My team is working on a large django project that incorporates multiple python apps (eventually installed on multiple nodes) and shared local libraries. While I already sold them on moving to poetry, getting a dev environment set up currently seems to more of a hassle than it's worth. I'd prefer to wait than use the work-arounds others have mentioned for simulating an editable install.

@abn I like your #2270 proposal and would like to help with any feedback/planning to get things rolling ASAP. My gitlab ticket of moving our project to poetry is going to hang until one of the proposed solutions gets implemented.

[marcelmindemann]

Would this work? I'd create a seperate pyproject.toml in a new folder which lists all of my main package's in-company dependencies as editable path dependencies. I then run poetry install followed by poetry shell. In this shell, with the venv activated, I then do my development in my main package. The dependencies in my main package's pyproject.toml can stay regular version dependencies, which allows poetry build. Since my shell already has a venv activated, poetry run commands will import the editable dependencies.

[misterjoa]

Hello everyone,
we are also facing this as we have a monorepo and want to be able to build dev packages locally and use defined versions of packages in prod.
Seems related to #2046

[janhurst]

I'm also facing a similar challenge working with a "main" application that uses a shared library that I have added as a Git submodule. Apologies in advanced I'm pretty new to poetry, coming from conda and manual venv.

I want to be able to work on the shared library, and interact with it's Git repo, but a CI/CD produced version of the shared library from my private Azure Artifacts feed when I build my application (in a Docker container).

I had expected behaviour similar to what I think the OP suggested where the local path was used when I poetry install when working in my dev environment, but a repo/feed version was used when I poetry install --no-dev in my Docker build.

Using something like:

[tool.poetry.dependencies]
python = "~=3.7.3"
my-lib1 = "^0.0.1"

[tool.poetry.dev-dependencies]
my-lib1 = { path = "./my-lib1-in-a-git-submodule/", develop = true }

I think this partially mitigates the issue the OP mentions about co-workers organizing their projects differently.... I have the luxury of "forcing" a project layout by using a Git submodule, but regardless I think this would still be useful?

[kesiraa]

How about having separate toml files for various environments. ?
i.e pyproject.dev.toml, pyproject.prod.toml
This will also allow us to test newer python versions (i.e 3.9 in dev to test, while 3.8 is in the prod)

[micahscopes]

If you haven't tried it, yarn has a nice feature for this called yarn link. It could be nice to have something like this in poetry!

[briggySmalls]

@janhurst I have exactly the same issue that you do. Our production docker containers use poetry to build a wheel before installing it in a fresh image. Currently the shared library in the dev-dependencies causes this build step to fail.

[mrijken]

I have made https://pypi.org/project/poetry-dev/ in order to support this use case.

[DanCardin]

I can see where this kind of request might get bogged down if you're creating potentially conflicting constraints with disjoint groups.

However for my usecase, which brought me to this issue, i feel like just deferring the install-time (poetry install) "resolution" of dependencies' until they're actually selected to be installed; would solve a/my/the problem.

And by "resolution", i mean that a path dependency will ensure the existence of the path before it checks whether or not it will ultimately be installed. Whereas for non-path dependencies, there really isn't any resolution of that sort until it's being installed (afaik).

If the lockfile is not outdated, the path must have existed at the time of the poetry lock. So the fact that it might not exist (e.g. docker builds) at install-time should be irrelevant, if it's going to be excluded for some reason (i.e. it being in a group being excluded by some specific install command (poetry install --only main)

EDIT: poet-plugin initially seemed like a solution, but it appears to not be invoked early enough to avoid the path existence check

[neersighted]

It seems like you have have come across the wrong issue -- you might be looking for #668 instead?

[DanCardin]

aha! dunno how i missed that. Although they do feel very very related. I'll go subscribe there instead :P

[neersighted]

To be clear to any onlookers:

• This issue tracks disjoint groups of dependencies, e.g. one version of the same dep in main, and another in dev.
• Do not validate path dependencies when they are not selected for installation #668 tracks skipping validating dependencies not selected for install, e.g. using an additional but non-overlapping dependency in development not present in the "main" group.

[joaoe]

Having IDE support for import and auto completion for files within the tests/ folder should "just" work.

That's a nice to have, but the main use case in my suggestion is running stuff from the terminal or CI pipeline, and stuff just works if they import dev-only dependencies, e.g., test files importing from other test files.

[TBBle]

I think there's two things being conflated there. A dev-only dependency, i.e., another distribution, will "just work" when it's installed, because it's installed like any other distribution, and added to sys.paths as described by its metadata.

If you want things in your tests/ folder to be importable, making them into a distribution with a setup.py as you have been doing, is feasible.

If you do not want to make that folder a separate distribution, then you would need to manually add it to your sys.paths, e.g., with PYTHONPATH, or a pytest, poe, or tox config option if using one of those runners.

I don't think this is something Poetry should handle "magically", because:

• there's no clear way to know when you want that test/ path added to sys.paths;
• this seems out-of-scope for Poetry as a package manager and build-tool;
• Poetry isn't involved if you're running other stuff from the terminal or CI pipeline.

This also seems like a very different concern from what this ticket is about, as summarised by #1168 (comment).

[rockandska]

Hi,

Was looking for a similar issue before opening a "feature request" and seems this one is related to what I was looking for.

Seems that poetry, even if in dedicated groups, build the dependencies tree based on all groups.
Seems fair since, depending on your needs, you install just "lint" group, or "formatter" group or two groups, with main, without main, run those groups with only one python version in contrary to the package itself etc...
But, sometimes, it constrains the users and hit dependency hell even if they use a group only for test/lint etc... or doesn't let them use a recent version of a package because one of the packages require an old version for a cross dependency.
As an example, flake8 and its dependency on importlib-metadata < 5

So, is there any chance to add something similar as depends_on = [] or exclusive = True for groups to let the lock file be built based on group dependency and let users have more flexibility on how the dependency is achieved ?

If not related with this issue and think it should be a separate "Feature Request", let me know.

Regards,

[TBBle]

Seems like a related request, since even though the original request was more focused on having different versions of the same library in different groups, e.g. one pathed at dev-time, and one semver'd at non-dev time, if the pathed version of the used library itself depends on a different set libraries than the published version, the locked package set is bifurcated based on dev/non-dev, which seems to be the crux of your use-case as well.

One concern with a bifurcated package set based on groups (which applies to dev/non-dev too) is that there's no way to represent that in a wheel generated by build, you just end up with a package where installing with both groups enabled will cause failure due to impossible-to-solve conflicts, which isn't super-kind to the user. This might be solvable by being able to exclude specific groups from the wheel metadata, as in your example, the group which pulls in flake8 could be marked to be ignored by build; that doesn't solve the problem with needing to support bifurcated locked dependency trees, but it does limit it to poetry install cases, where Poetry is able to be involved and tell the user "These two groups can't be installed together" from its own metadata.

[hwmq]

I have an issue that at first I thought was relevant to this ticket, but now I think not so much. It may just be a niche use case if this ticket were resolved. It is probably like a feature request for poetry-core and/or pip. I'll delete this comment later.

I have a core library and a cli library. cli depends on core and declares core in tool.poetry.dependencies as path = "/path/to/core", develop = true. I would like developers to be able to make an editable install of cli using pip (for reasons I'd rather not have to clarify; again, maybe niche).

When I pip install -e /path/to/core, I get an editable core in my user site-packages. Great.

Obtaining file:///path/to/core
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Installing backend dependencies ... done
  Preparing editable metadata (pyproject.toml) ... done

But when I pip install -e /path/to/cli, the editable core is removed and a non-editable core is installed instead.

Obtaining file:///path/to/cli
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Processing /path/to/core (from cli==0.0.0)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done

The workaround is pretty simple: editable install cli first, then editable install core.

I hoped that when I ran pip install -e /path/to/cli, my core dependency would either be detected as satisfied by the existing editable core install, or just be built and installed as editable again because it is marked as develop = true.

It would be nice if I could have core declared as both a non-local dependency for production, and a local path dependency for development, like how this issue describes. If I request a non-editable install with pip I might hope the non-local dependency is selected, and if I request an editable install with pip I might hope the local path dependency is selected (and built+installed as editable as well). I think this might depend on the get_requires_for_build_editable hook being able to return a local vs. non-local requirement?

To disclaim again, I'm aware this may be too opinionated/unintuitive/niche, or that I am expecting too much coordination between pip and the build backend.

[TheKevJames]

Some more thoughts on this in an issue which I just closed as "mostly a dupe", for those collecting more info, somewhat specific to the non-dev dependency having a specified source: #8885

[luigimannoni]

⚠️ after a couple of months dealing with the issue the below workaround does not cope well with version pinning and generated lock files, sometimes will lead to weird behaviours in poetry. I'll keep it for record, but avoid. ⚠️

I've spent days with this one, as I had a setup of 5 different libraries with intricate dependencies one to another all based on a vcs link for production deployment, that meant the libraries needed to release before I could test and play with code on the other ones and if I needed to adjust the dependency meant another release.

Now I had the "pleasure" to centralise everything into a single repo with multiple git submodules and Dockerize the entire workflow to install automatically everything, take care virtual envs and make the libraries editable without polluting too much the host system.
I tried different tomls but it's frowned upon on #4460 (fair enough reasoning), and it's impossible to don't mess it up when updating dependencies versions. Then it struck me, just use an optional group for production dependency, for example:

My root ./pyproject.toml, of where I can import and play with different code

[tool.poetry.dependencies]
python = "3.10.*"
library-1 = {path = "library-1", develop = true}
library-2 = {path = "library-2", develop = true}
library-3 = {path = "library-3", develop = true}

Library 2 ./pyproject.toml:

[tool.poetry.dependencies]
python = "3.10.*"
dep1 = "^1.0.0"
dep2 = "^1.0.0"
...

[tool.poetry.group.dev.dependencies]
library-1 = { path = "../library-1", develop = true }

[tool.poetry.group.production]
optional = true

[tool.poetry.group.production.dependencies]
library-1 = { git = "ssh://[email protected]/mygit/library-1.git", rev = "v1.0.1" }

Library 3 ./pyproject.toml:

[tool.poetry.dependencies]
python = "3.10.*"
dep3 = "^1.0.0"
dep4 = "^1.0.0"
...

[tool.poetry.group.dev.dependencies]
library-1 = { path = "../library-1", develop = true }
library-4 = { path = "../library-4", develop = true }

[tool.poetry.group.production]
optional = true

[tool.poetry.group.production.dependencies]
library-1 = { git = "ssh://[email protected]/mygit/library-1.git", rev = "v1.0.1" }
library-4 = { git = "ssh://[email protected]/mygit/library-4.git", rev = "v1.1.0" }

etc...

Then on CI for each library I can run poetry install --without dev --with production --no-interaction --no-ansi --no-cache and it gets the remote revs, on local a simple poetry install --with dev uses the local editable libraries.

Of course I still need to take care to update revisions and version numbers across the tomls as waterfall before releasing, but that can be done with a script and in any case part of a normal workflow anyways.

Yes the big caveat is that people need to be able to modify the sub-projects tomls and still can't do overrides on sub-dependencies dependencies like composer does in php but that can get control back on multirepo libraries under the same ownership.

[jmdeschenes]

I've spent days with this one, as I had a setup of 5 different libraries with intricate dependencies one to another all based on a vcs link for production deployment, that meant the libraries needed to release before I could test and play with code on the other ones and if I needed to adjust the dependency meant another release.

Now I had the "pleasure" to centralise everything into a single repo with multiple git submodules and Dockerize the entire workflow to install automatically everything, take care virtual envs and make the libraries editable without polluting too much the host system. I tried different tomls but it's frowned upon on #4460 (fair enough reasoning), and it's impossible to don't mess it up when updating dependencies versions. Then it struck me, just use an optional group for production dependency, for example:
....

This is pretty nice!

Unfortunately, if you use dependencies from pypi, it won't work:

[tool.poetry.group.production.dependencies]
library-1 = { version = "0.1.0", source = "my-source }

Maybe poetry should not check for this?

[jonapich]

I wrote some tooling to help with the use case... Sorry it's using the dev-dependencies thingie:

• published library with local path for dev
• monorepo virtual environment

It works pretty well. The key for the root dev project is this pydev=true.

The tooling is called stew. The relevant commands for the use case:

• stew bump will bump all lock files in the monorepo
• stew pull-dev-dependencies will analyze the dev dependencies of all the projects and import them into the root's pyproject
• stew check-outdated will report on outdated lock files, or outdated dev dependencies.
• stew build is similar to pip download, it bundles everything so you can install offline. Docs here.

I'm not sure it works with the new poetry groups at all, by the way. It's also opiniated around git.

[KirillShoniya]

Same issue for me =-(

I develop project with few services.

Some of this services depends on our library (shared-lib).

In develop process we need to do some changes in that library in editable mode. For better development experience.

So I tried solve the problem in this way, but it's not working:

[tool.poetry.dependencies]
python = "3.11.*"

[tool.poetry.group.shared.dependencies]
shared-lib = {version = "0.0.1", source = "private-registry"}

[tool.poetry.group.dev.dependencies]
shared-lib = {path = "/path/to/shared-lib/", develop = true}

I got an error:

Incompatible constraints in requirements of my-project (0.0.1):
shared-lib @ file:///path/to/shared-lib (0.0.1)
shared-lib (==0.0.1) ; source=private-registry

This could be used like this:

# local
poetry install --without shared --with dev

# deployment
poetry install --without dev

Is any solutions for this case?

[aman-gupta-doc]

Hi

Any new update or development for this?