feat: Support pyproject.toml

[olivier-lacroix]

Hi there,

A (very) rough and partial implementation of pyproject.toml support for pixi.

• A kind field is added to the Manifest struct to differentiate between the pixi.toml and pyproject.toml
• The kind is detected from the (potentially discovered) manifest file
• the pyproject.toml is parsed, reading the tool.pixi part(s) as it would the content of pixi.toml
• pyproject project.requires_python and project.dependencies are injected into the Manifest as conda dependencies, unless they already exist in tool.pixi.dependencies or tool.pixi.pypi_dependencies

Keen to get some early feedback before I progress any further on your interest in such a feature and on the proposed approach.

Would be further required, among others:

• Handling of the addition / removal of a dependency
• Handling of extras, mapping them to features & environments

[ruben-arts]

Hello @olivier-lacroix and everyone following this.

The team is very positive on the support for pyproject.toml and we love the work you put in to prepare a first version.

Its however not a small feat to support a second manifest. Next to this PR we would also need to support file modification for the cli commands like pixi add and pixi remove.

So we would like to take some time to design it properly.

Meanwhile we would like you to continue on this PR and get it tested. I will support this PR and start testing right away.

To the community: I would like to ask you to start testing this PR if you can!

Again thanks for the work, we really appreciate the work from the community and hope we can get this feature to a next level, together ❤️!

[olivier-lacroix]

Thanks @ruben-arts !

Keen to get your feedback on the directions / choices made here as I do not know the codebase very well. It's also my first chunk of Rust code, so pretty sure things can be improved :-).

[ruben-arts]

Some initial thoughts, we should also add a test to make sure we support the pixi full spec in pyproject.toml. It's kinda inherited because you just read the toml table prefixed with tool.pixi but it is a good test to make sure.

[baszalmstra]

I like where this is going! But I think we need to refactor the structure a little bit more, let me explain:

Currently the manifest::Manifest contains a kind and a toml_edit::Document. I would opt to move both of these fields into an enum that represents and editable source of the Manifest. We have a lot of functions in Manifest that modify this document but depending on whether the source is a pyproject.toml or a pixi.toml this will need to behave differently. To adhere to the single responsibility design I think it makes sense to move this into its own type.

e.g.:

pub enum ManifestSource {
    PyProjectToml(toml_edit::Document),
    PixiToml(toml_edit::Document),
}

[olivier-lacroix]

@baszalmstra I took your comments into account.
@ruben-arts do you have such a pixi.toml handy I could convert into an equivalent pyproject.toml?

[ruben-arts]

@olivier-lacroix In theory all pixi.toml should be convertable to a pyproject.toml right? I might not be reading your question correctly.

[olivier-lacroix]

Yes @ruben-arts . ButI understood you wanted a test with a file that had all pixi features?

[ruben-arts]

Ah sorry I see, @olivier-lacroix. You could check schema/examples/valid/full.toml, this should include most of it.

[ruben-arts]

He @olivier-lacroix, your going very fast, awesome!

We've taken a good look at how we would want to implement it. The only big change for now is that we want to import the project.dependencies as pypi-dependencies as we cannot trust the map between conda and pypi.

We're currently working hard on supporting pypi source dependencies. Which we also want to mix with this feature where we should install the actual project itself aswell using pixi install if we are using a pyproject.toml as our pixi configuration. Which will hopefully greatly improve this experience.

[ruben-arts]

In my opinion it is ready for review! Awesome work @olivier-lacroix

[olivier-lacroix]

Thank you @ruben-arts !

[ruben-arts]

@olivier-lacroix Added some documentation, let me know if it looks good.

[pavelzw]

Hmm, i'm wondering whether we should make this behavior implicit. Currently it's always running with manifest-path $PWD/pixi.toml

EDIT: let's do this discussion in prefix-dev/setup-pixi#87

[olivier-lacroix]

Thanks a lot for that @ruben-arts !!

[pavelzw]

Just to be sure: when specifying both [tool.pixi] in pyproject.toml and pixi.toml, pixi would fail?

[olivier-lacroix]

Just to be sure: when specifying both [tool.pixi] in pyproject.toml and pixi.toml, pixi would fail?

Not sure what you mean @pavelzw ?

[pavelzw]

When we basically define the pixi configuration two times, once in pixi.toml and once in pyproject.toml. Does pixi handle this error case or just silently ignore one config?

[olivier-lacroix]

It will pick pixi.toml in priority, and ignore the pyproject.toml. The same way the discovery for pixi.toml up the folder tree stopped at the first pixi.toml found.

[ruben-arts]

I would be in favor of having this behavior as imo conda packages

Do you(@pavelzw) mean you want the automatic mapping @olivier-lacroix describes in: #532 ?

[olivier-lacroix]

Hi all, let me know if you expect me to add anything else to this PR before merging.

re. the presence of both pixi.toml and pyproject.toml, let me know if you want pixi to fail, or if documenting the current behaviour is enough (I feel like the current behaviour is OK, but that's just me :-) )

[baszalmstra]

Some small doc changes, but otherwise looks good to me!

[ruben-arts]

Thanks a lot @olivier-lacroix!

[olivier-lacroix]

Great! thank you all for your guidance & feedback!

[ruben-arts]

@olivier-lacroix we are working on a blog-post where we are also going to mention your work on pyproject.toml. But I'm not sure how to mention you, to give credit where credit's due.

[olivier-lacroix]

@ruben-arts , that's very kind of you and also quite unecessary :-). my github handle is not very creative and is my firstname-lastname

[ruben-arts]

@olivier-lacroix will mention your GitHub then! If you have other socials let us know!