Add support for app/library type to poetry new and poetry init

[edmorley]

Issue Kind

Brand new capability

Description

Currently if I use poetry new or poetry init, I end up with a pyproject.toml configuration that is more oriented towards a library than an app, with no way to tell the new / init commands otherwise (beyond --python, but that's only one of several things that would need overriding).

For example they generate:

[tool.poetry]
name = "example"
version = "0.1.0"
description = ""
authors = ["..."]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

...whereas for an app:

• it's rare to want to publish the app (ie: I want package-mode = false from Introduce non-package-mode #8650, and don't want the name/version/description/authors/readme fields)
• it's rare to need to build the app (ie: I don't need the [build-system])
• it's safer to set a specific major Python version (eg 3.12.*) rather than allowing an unsafe range (such as >=3.12) - so prod vs staging vs each developer's machine is at least using the same major Python version.

And therefore for apps, a config like the following is more appropriate:

[tool.poetry]
package-mode = false

[tool.poetry.dependencies]
python = "3.12.*"

As such, it would be helpful if the poetry new and poetry init commands accepted something like a --type argument (and corresponding question prompt when using those commands in interactive mode), that accepted options like "app" or "library", that then generated config more appropriate for the specified use-case. The interactive prompts and/or existing CLI args would still allow users to mix and match if needed (for example, the user could select "app" mode, but when prompted for the python value, could override the app mode's suggested default of "3.N.*" to a wider range if they prefer.

Such a type option would be similar to:

• uv init's --app vs --lib: https://docs.astral.sh/uv/reference/cli/#uv-init
• cargo new's --bin / --lib: https://doc.rust-lang.org/cargo/commands/cargo-new.html

Possible arg/option names:

• --type {app,lib} (or --type {app,library}
• --app / --lib (or --app / --library)

Note: I intentionally didn't include --package-mode {true,false} in the list, since this feature request is about more than just package-mode = false and about several other defaults that make less sense when using Poetry with an app.

Impact

• Means users can more easily generate a config that contains best practices for app use-cases, rather than them having to discover additional config/best practices via the docs and change it retrospectively
• Likely means fewer support tickets about things like "why is Poetry saying I have to have a README" etc, since not all users read the docs properly
• Encourages projects which are apps to not use unsafe unbounded Python version ranges (which can cause breakage in CI/CD environments when new Python versions are released, or cause user confusion when production works differently from locally when they happen to be using different Python versions that are permitted by the wide range)

Workarounds

Users either:

• avoid using poetry new / poetry init at all, and create their pyproject.toml from scratch using the docs
• use poetry new / poetry init and fill out questions that aren't relevant, but then have to change the defaults afterwards

...but both of these rely upon the user knowing that the default configs generated by new/init are not ideal for app use-cases, and what they should change to make them more suitable.

[edmorley]

Encourages projects which are apps to not use unsafe unbounded Python version ranges (which can cause breakage in CI/CD environments when new Python versions are released, or cause user confusion when production works differently from locally when they happen to be using different Python versions that are permitted by the wide range)

For more on the issues this causes, see:
heroku/buildpacks-python#260

[radoering]

In my opinion this distinction between app and lib is flawed for Python projects. (There are many apps that are packaged, e.g. Poetry itself, pipx, pre-commit, ...)

Setting package mode as in #9622 and maybe doing more based on this makes more sense to me, because:

• in non-package mode you cannot build so you do not need a build system
• readme and other fields are not required (cf Add package-mode option to init and new commands #9622 (comment))

it's safer to set a specific major Python version (eg 3.12.*) rather than allowing an unsafe range (such as >=3.12) - so prod vs staging vs each developer's machine is at least using the same major Python version.

I have not thought about that but we could consider to do this when creating a project in non-package mode, too.

[edmorley]

Thank you for the reply! :-)

In my opinion this distinction between app and lib is flawed for Python projects. (There are many apps that are packaged, e.g. Poetry itself, pipx, pre-commit, ...)

Yeah I definitely agree that there are types of projects where it's a grey area between "app" and "library" (eg CLI tools, Poetry itself etc).

Naming is hard and all that...but by "app" I was instead meaning ~"end-user" projects that are typically not distributed for the world to use, but are used directly by the person or team maintaining them for a very specific purpose. Think Django webapps that people are deploying to production, or data analysis projects shared by a team of data scientists etc, rather than an open source project that's shared for consumption by others.

I would argue these types of "app" projects:

1. Are much more common (by quantity of projects) than libraries or the "in-between an app and a library" type tools like CLIs/Poetry/pipx that you mention.
2. Typically will only ever be run/tested using a single version of Python.

Given (1), I think it would be really beneficial for Poetry to support these types of projects better out of the box. (Plus, I would argue that the authors of libraries or of tools like pipx are much more capable of overriding defaults than most end users, so it makes sense to pick the defaults for the more-common and more-beginner-heavy app project type instead.)

The current default tool.poetry.dependencies.python value of e.g. >=3.10 is counter-productive for these projects, since they typically aren't tested against several Python versions in a CI matrix. So using anything other than the same version locally/across different developer machines/in production etc is asking for trouble.

I would also posit a big reason these types of projects use a package manager like Poetry (instead of using pip and requirements files), is because they want the determinism of a lockfile. However, without also ~"locking" the Python version to at least a major version (eg ==3.12.*) then they can still encounter the classic "works on your machine but not mine / in production".

As-is, all of the options for us (Heroku) adding support for reading the Python version from tool.poetry.dependencies.python aren't great - unless Poetry were to start encouraging "app" type projects to use safer/more-app-appropriate version ranges (such as ==3.X.*) by default. For more on why, see:
heroku/buildpacks-python#260

In contrast, uv's behaviour is already a lot more suitable out of the box for these app type projects: The uv init command creates .python-version file at the same time as populating pyproject.toml, and whenever any uv command is run validates that the .python-version file and requires-python fields are still compatible. There are still some improvements I'd like to see (xref astral-sh/uv#7429 (comment)), but right now uv is ahead in this area for ease of app deployment, and it would be great to close the gap.

[dimbleby]

I expect that, although you want to force apps to be not portable to different pythons, other users would find that an annoying default.

Specifically wildcarding the python version is error-prone, let's not make that any sort of default #8418. Prefer an upper bound, if you must.

However: putting a very narrow range in pyproject.toml feels to me as though it is conceptually probably not the right semantic. That file is about the abstract project requirements, locking a particular concrete environment happens in poetry.lock.

Except of course that poetry does not lock a python version. I think uv may be different here, perhaps that is significant. (Aside: if you like uv - or any other tool - better - then use that!)

IMO if you want a very narrow python version some out-of-band (from poetry's point of view) mechanism is the way to go.

[edmorley]

Thank you for the reply.

I expect that, although you want to force apps to be not portable to different pythons, other users would find that an annoying default.
...
Except of course that poetry does not lock a python version. I think uv may be different here, perhaps that is significant. (Aside: if you like uv - or any other tool - better - then use that!)

This isn't about liking uv better. This is me giving feedback as the maintainer of a PaaS adding support for Poetry (alongside our existing pip + Pipenv support), who has a lot of contact with real world deployed projects and common user pain points.

Poetry's current defaults/design choices are currently very skewed towards the use-cases of packaged/published/widely-distributed type projects - which is a project type that is much less common than the simple "app that doesn't need to be published and probably doesn't even need a build backend" case. If Poetry wants to increase user adoption (or reduce attrition to uv), then it seems a few tweaks to cater for this more common project type might be worth considering?

However: putting a very narrow range in pyproject.toml feels to me as though it is conceptually probably not the right semantic. That file is about the abstract project requirements, locking a particular concrete environment happens in poetry.lock.

But for a simple app that isn't being distributed, the "project requirements" are actually that it only support a single Python version. It having to declare that it supports multiple is both unnecessary and likely a lie, given the lack of CI etc (described in #9668 (comment)).

That said, an alternative option would be for Poetry to do what uv does, and for poetry {new,init} to create a .python-version file with a resolved Python version at the same time as generating the pyproject.toml (in uv's case, it adds just the major version to .python-version, so eg 3.12).

The downside to this is that for the common "simple app that isn't distributed" case, the user now has to update/maintain the Python version in two places (.python-version and requires-python), and if they leave requires-python as a wide range (eg >=3.10), then it means the resolver is unnecessarily restricted in what package version it can pick (especially relevant as Python versions are EOLed; xref the "Lastly, imagine the scenario where" section in astral-sh/uv#7429 (comment)).

[dimbleby]

If Poetry wants to increase user adoption (or reduce attrition to uv)

"poetry" doesn't want anything, people who work with it may want different things. So far as I am concerned - I am very comfortable with people choosing whichever tool works best for them. (As per the comment that you are replying to!)

I understand that you have a use case where you want to specify a narrow python range (though apparently still not an exact version) Perhaps we are just repeating ourselves now but summarizing my last: I suspect this is not so universal use case as you think, and I reckon this is not a great fit for being solved in poetry.

[edmorley]

I suspect this is not so universal use case as you think

I think this is the primary point of contention :-)

I would posit:

1. Of all projects using Python, there is at least an order of magnitude more projects that are never published to PyPI or distributed to people who aren't the maintainers of the project, than those that are.
2. That for this group of projects:
   1. The majority will not be tested in CI against multiple Python versions (and so no guarantee they would work on any other Python version other than the single version that was run locally or in CI).
   2. That if the authors of the project had to choose between determinism/reliability and ability to run on multiple Python versions, they would choose the former. (In reality, many beginners won't know they are making this choice; which is why defaults matter.)
   3. And therefore that running the project on multiple Python versions is for them a bug not a feature (think error in prod since staging or local development Python version differed, or lack of parity between local development environments on different machines).

and I reckon this is not a great fit for being solved in poetry.

Is Poetry primarily focused on people who package/publish/distribute projects? If so, then yeah perhaps Poetry just intending to be a good fit for these "end user app" type use-cases.

However, the current README / https://python-poetry.org descriptions seem to suggest it's also aimed at these use-cases, which is perhaps where my confusion lies?

[dimbleby]

offering a datapoint: I have a django app that I deploy to Azure, and the definition of the Azure web app is where I specify the version of python that the deployment uses.

It's been a while since I used similar services eg heroku but I seem to recall - and would expect - that they too offer a knob where one specifies the deployed python version.

In local development I might or might not be using the same python version that I deploy with.

so I prefer to have a relatively broad range in my project definition, and only have to update the configuration in Azure (or wherever) when I want to upgrade. Defining a narrow python version in two places would be a small step backwards, rather than forwards.

[edmorley]

It's been a while since I used similar services eg heroku but I seem to recall - and would expect - that they too offer a knob where one specifies the deployed python version.

I maintain Heroku's Python support, hence starting this discussion :-)

We want to move away from platform-proprietary ways of specifying the Python version, and be able to reliably determine what Python version to use when deploying a project from a canonical version used by other tooling (e.g. .python-version, or failing that, fall back to requires-python or tool.poetry.dependencies.python etc).

That doesn't mean there isn't still value in having a way to override the version for Heroku specifically if needed (similar to you adding config to your Azure app definition), but that "deploying the same Python version you tested with" is a pretty fundamental concept for reliably deploying arbitrary end user code, and it's beneficial to not force users to add yet more platform-specific config in the common case.

In local development I might or might not be using the same python version that I deploy with.

Based on my experience seeing (and having to deal with) the issues end users run into (at Heroku scale, so not a small dataset), I would confidently state that using different Python versions for local development vs deployment is more often than not an anti-pattern. This pattern should still be permitted of course, however it should not be encouraged by the defaults used by beginners - and instead be something experienced like yourself can opt into. Defaults really do matter!

[dimbleby]

We want to move away from platform-proprietary ways of specifying the Python version, and be able to reliably determine what Python version to use when deploying a project from a canonical version used by other tooling

well now we're getting to the heart of the matter.

This fits with my comment about the poetry configuration not conceptually being the right thing here. To my mind this is simply not what that field is for, and it is overloading the poetry configuration to use it that way.

I think I've probably said all that I have to say here. Suggest that if you want more opinions then the most likely way to flush them out would be to submit a merge request, it doesn't sound as though it ought to be very much effort.

[edmorley]

This fits with my comment about the poetry configuration not conceptually being the right thing here. To my mind this is simply not what that field is for, and it is overloading the poetry configuration to use it that way.

That's a fair point. (Albeit I do believe that semantically it is still correct to say many of these projects only support a single Python version, since they aren't tested against other versions.)

Regardless, I still believe there would be a benefit to Poetry helping with the "control/lock the Python version" part of the project management UX here. Ultimately tools like pyenv, GitHub Actions, IDEs, PaaS deployment workflows all need to pick a single Python version to install, and an unbounded version range doesn't let them reliably pick a version.

Perhaps the solution is to double down on .python-version as the canonical way to do that (along with some changes to help reduce the pitfalls of having the version specified in two places)?

For example, by:

1. Adding optional support to poetry {new,init} for creating a .python-version file (similar to what uv does).
2. Adding a --type {app,lib} option to poetry {new,init}, whereby type=app would enable the .python-version file creation, and configure pyproject.toml to use package-mode = false and omit the license/readme/authors fields.
3. Making Poetry warn (or error) if the Python version in .python-version isn't compatible with the range set in pyproject.toml (to at least help prevent the two version sources getting out of sync).
4. Have Poetry warn if the Python version range set in pyproject.toml includes an EOL Python version. (This would help with the scenario where users only ever update .python-version, the version in pyproject.toml becomes increasingly stale, and then it negatively impacting package resolution as Python versions EOL over time).

I'm happy to file separate issues for 1+3+4 (2 is covered by this issue already, with a tweak to the Python version parts) if they sound reasonable to you?

[dimbleby]

honestly I don't love any of this, but I am only a contributor and in the end my vote doesn't matter anyway.

As a matter of tactics if I were trying to get change accepted, I think it's reasonable to expect that smaller changes have a better chance than larger changes. Also making smaller changes first allows you to test the water. So if you see a sensible way to split things up, that sounds reasonable to me.