Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for pixi lock files #1339

Open
matthewfeickert opened this issue Mar 28, 2024 · 1 comment
Open

Add support for pixi lock files #1339

matthewfeickert opened this issue Mar 28, 2024 · 1 comment

Comments

@matthewfeickert
Copy link
Contributor

conda-lock itself hauls in... a lot of dependencies, so may not be a good candidate for the "base coat" environment. micromamba, already present, is certainly up to the task of consuming both formats... though pixi very well might end up "winning" for this use case.

Originally posted by @bollwyvl in #1312 (comment)

This was already touched upon in Issue #1312 a bit (in the above), and now also with conda/conda-lock#615 being open, but it would be quite nice if pixi lock files could be supported formally by repo2docker. The ability to do cross-platform solves in parallel already brings the accessibility barrier for projects down quite a bit, and then being able to reuse that environment solve for repo2docker is very attractive from a user perspective.

I appreciate that supporting a new standard is a lot fo work, and @bollwyvl has already shown in Issue #1312 that these sorts of changes are hard. I'm asking as I think it would be very cool!

@minrk
Copy link
Member

minrk commented Aug 22, 2024

I've been thinking about lock files on and off for a long time, specifically about how to reconcile a "fully specified" environment with a lockfile with the repo2docker approach of a 'base' environment with jupyter-server, etc. about which we have some assumptions for how to launch, which gets updated with a user environment. The problem generally is that repo2docker's design is meant to enable folks to write environment specifications that don't consider Jupyter, but we make it work by adding the required dependencies. This isn't friendly to using lockfiles (and is generally a problem class of its own).

There are 3 basic approaches to this that I can think of:

  1. install base env, then 'update' with lock, like we do currently with 'loose' environments (could cause conflicts, doesn't really satisfy lock specification)
  2. install lock first, then try to minimally add 'required' packages (doesn't satisfy lock specification, may break things in different ways)
  3. leave lock unmodified, in which case required packages may be absent

So far, I think the best approach is the last one, to respect lockfiles exactly as-is, and install nothing 'automatically', with the following logic:

  1. inspect lockfile, and if jupyter-server is present (or specify some other condition), use it as the default env
  2. if jupyter-server is not present, install a kernel-only env, like we do for 'old' Pythons (< 3.7).

The case this will not handle is if no kernel package is in the lockfile, there will be no kernel installed in the locked env. I don't think there's any way for us to install ipykernel that won't make a mess at least sometimes. Usually pixi add ipykernel would work! But it's a lot harder to be sure and a lot harder to reason about "your lockfile is modified slightly sometimes" than "your lockfile is respected, but you might need to add packages to it for it to work with repo2docker".

This approach would be generally applicable to lockfiles (conda-lock, pixi, anything else we decide to support). There are trade-offs for everything, but I think this is the one that's easiest to deal with and most consistent with what lockfiles are meant to do.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants