Python: parse conda.env in addition to requirements.txt

[RXminuS]

Right now Depandabot can only deal with requirement.txt files from PyPy but a lot of our repositories use Anaconda yaml files because of the performance increase with the Tensorflow installation. Thus, adding Conda yaml support would be essential for us!

[greysteil]

Interesting - are there any open source repos you can point me to that use that setup (that I can use as fixtures when I write this)?

[RXminuS]

https://github.com/fastai/fastai the environment.yml (or environment.yaml) It's fairly popular in ML/AI communities to do it like this

[gr33ndata]

I second that, it would be useful to update both requirements.txt and environment.yml

[palfrey]

I'm using Conda requirement files (https://github.com/palfrey/waveform-necklace/blob/master/conda-requirements.txt) and hitting issues (see palfrey/waveform-necklace#19)

[oji]

We use conda as well, with the environment.yml variant. Is there any plan to add support for this? Thanks!

[rebelagentm]

@palfrey @oji Thank you! We think this will be useful and agree on supporting it, but the team is stretched thin right now as we work to integrate and scale Dependabot into GitHub.

If you're keen to help us add support, we would be happy to review any pull requests for it on dependabot-core.

[alexvicegrab]

Would be useful to have.

[epruesse]

This would be very interesting and useful to have.

Some notes:

• conda is a general purpose package manager not limited to Python
• anaconda cloud is the default service for hosting packages
• miniconda is the minimal distribution including the package manager
• channels are collections/distributions of packages. Anyone can create one, by default hosted at anaconda cloud, but not necessarily so.
• Common channels besides the basic defaults it comes with conda-forge for general purpose software and bioconda for life science specific packages.
• There is (currently) no standard for placing the environment.yaml. The file format really just describes an arbitrary selection of packages one might have in a conda "virtual environment".
• A more specific file would be recipe/meta.yaml, which contains be the package build instructions (different format), including dependencies split into build, host and run (where build vs host separates the build system from development libraries in the case of cross compilation).

[brl0]

I too think this would be great.

The community around this package manager (and some related ones) really seems to be growing and blossoming. I think it is also worth noting that while many conda related packages and resources are python related, it is not strictly a python package/environment manager, as it also supports Python, R, Node.js, Java, among others.

Not sure if this is useful, but here are some possibly relevant resources.

I would think the conda-forge community, probably the core team in particular, would be the most authoritative resource (besides the original developers Anaconda/ContinuumIO).

I think it is worth noting the derivative or related package managers:

• miniconda: Miniconda is a free minimal installer for conda
• miniforge: minimal installer for Conda specific to conda-forge
• mamba: reimplementation of the conda package manager in C++

Another related project of interest is conda-lock, which is hosted by the conda-incubator organization.

Libraries.io bibliothecary (also written in Ruby) supports conda (see here) using a simple python web service conda-parser.

Also recently saw a couple of interesting and possibly loosely relevant packages in the Regro GitHub organization, which is a sister organization to conda-forge. In particular, these repositories do a lot of work mapping conda dependencies:

• https://github.com/regro/cf-scripts
• https://github.com/regro/libcflib

Binderhub is an open source project and service that allows users to share reproducible interactive computing environments from code repositories, and it can make use of conda environment files via repo2docker. I think that adding conda support to dependabot would help make it easier to maintain and support these environments.

Hopefully some of this is helpful or interesting and helps motivate adding conda support.

[mrietberg]

@rebelagentm Any update on the support for Conda?

[rebelagentm]

@mrietberg 👋🏻 I'm no longer working on Dependabot. Tagging @asciimike here, as he may be able to provide an update.

[asciimike]

We don't currently have plans to add support for Conda (tracking 👍🏻 on new-ecosystem requests is a rough way to gauge our prioritization), though we'd be happy to review PRs that the community creates.

[1kastner]

Some notes:

• conda is a general purpose package manager not limited to Python

• [...]

• There is (currently) no standard for placing the environment.yaml. The file format really just describes an arbitrary selection of packages one might have in a conda "virtual environment".

• A more specific file would be recipe/meta.yaml, which contains be the package build instructions (different format), including dependencies split into build, host and run (where build vs host separates the build system from development libraries in the case of cross compilation).

Here we might need to differentiate between conda packages that are meant to be installed and work material that is meant to be handed out for some event or as supplementary material for a publication. When I prepare seminar notebooks, I do hand out an environment.yml for a quick setup but I would never craft a recipe/meta.yaml. When we look at the repositories that are part of literature (such as introduction to ML etc.), we also rarely see this. In my opinion, these recipes are reserved for standalone packages that other people might import (such as a Python module) or use from the CLI (actually any kind of software). Now, should these people be forced to use recipes anyway?

[thomas-bc]

Any progress on this? 🙏

[pavelzw]

Maybe it could make sense looking into pixi. They use the conda universe but in a somewhat structured way. Pixi projects always use a pixi.toml or pyproject.toml and have a pixi.lock which contain the lockfile. Somewhat similar to poetry but for conda.
Probably it's easier to implement this into dependabot since this has a clearly defined project structure.

[jtroe]

Is this on the roadmap at all? This would be a welcome enhancement for my work at @Esri.

[Zeitsperre]

Another comment here that this would be immensely helpful to have. Thanks!

[isaacsanders]

https://github.com/conda/conda-lock is the main locking mechanism for people who use lockfiles in conda, I think.

[pavelzw]

nowadays pixi is: conda/conda-lock#615

[jezdez]

nowadays pixi is: conda/conda-lock#615

Hi all, conda steering council member here.

I think that's a bit of a misrepresentation of the facts. The linked issue discusses how conda-lock could position itself to pixi, which implements an own locking mechanism incompatible to conda-lock.

I believe the conda-lock maintainer has decided that it would become a kind of backend for conda-lock, in addition to the existing formats. Pixi has not supplanted conda-lock or become the default. See the upcoming conda-lock 3.0 release.

As context, conda-lock is, for better or worse, a conda community project with approval from the conda steering council, while pixi is a commercial project by Prefix.dev. I think that it matters to state this in a consequential feature request like this here.

[beckermr]

Thanks for the comment @jezdez. I am another conda steering council member and 100% agree.

[ytausch]

pixi is a commercial project by Prefix.dev.

Since pixi is licensed under BSD-3-Clause, I think "commercial" is a bit of an overstatement here.

[xhochy]

pixi is a commercial project by Prefix.dev.

Since pixi is licensed under BSD-3-Clause, I think "commercial" is a bit of an overstatement here.

The difference here is the controlling entity. conda-lock is part of the conda organisation and pixi is controlled by a single entity. The two tools also cater to a different workflow/developer UX. Thus it would make sense to support both of them.