Unexpected behavior change from pixi v.0.18 to v0.19 for github pypi dependencies

[langevin-usgs]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

Issue description

We have recently switched our MODFLOW continuous integration approach to use pixi. We had it set up and working with pixi version 0.18 and things were working well. Our CI workflow started failing today in response to a change introduced to version 0.19. This is the error message we see:

Run prefix-dev/[email protected]
  with:
    pixi-version: latest
  env:
    PIXI_BETA_WARNING_OFF: true
Downloading Pixi
  Pixi installed to /home/runner/.pixi/bin/pixi
Restoring pixi cache
pixi install --locked
  /home/runner/.pixi/bin/pixi install --locked --manifest-path pixi.toml --color always
    × lock-file not up-to-date with the project

The problem stems from the use of github pypi dependencies in our pixi.toml file.

[pypi-dependencies]
flopy = { git = "https://github.com/modflowpy/flopy.git" }
modflowapi = { git = "https://github.com/MODFLOW-USGS/modflowapi.git" }
modflow-devtools = { git = "https://github.com/MODFLOW-USGS/modflow-devtools.git" }

The pypi dependency hashes in pixi.lock are not up to date. Running pixi install --locked with 0.18 succeeds, and pixi install does not update locked hashes. With 0.19, pixi install --locked fails with the error message above, and pixi install updates the hashes.

It's unclear to us why pixi 0.19 is issuing the "lock-file not up-to-date with the project", whereas pixi 0.18 did not. I'm working on this with @wpbonelli and @Hofer-Julian.

Expected behavior

pixi version 0.19 should behave the same as 0.18 w.r.t. github pypi dependencies.

[ruben-arts]

He quick reply from my phone, this was actually a fix in v0.19.0, your lockfile is probably incorrect because it misses purls.

[ruben-arts]

Running pixi install with v0.19.0 should fix the lockfile. And should be backwards compatible with v0.18.0.

[wpbonelli]

Thanks @ruben-arts I take it #1148 is the fix you are referring to. I wonder if you would help clarify some things.

If we update the lockfile as you suggest, will the setup-pixi action fail (as it does now) whenever a github dependency is found to be out of date? Do we need to set frozen to avoid this, since the action defaults to locked if a pixi.lock file is found?

How would you suggest proceeding if we don't want to lock github dependencies? Historically we have elected to depend on github branches for projects which are developed more or less in lockstep with MODFLOW 6. Is there a post-install hook or some way to install them on pixi install? Or is a custom task (that must be run separately) the only way?

[ruben-arts]

That is indeed the fix.

I'm not sure why the GitHub repos would trigger the invalid lockfile. I'll have to look into that, as we don't check if the revisions in the lockfile are up-to-date in the --locked check.

We implemented the same behavior as cargo. So I guess we should create an update command so you could do pixi update pkgx pkgy pkgz on a ci run if you want that.

[tdejager]

I think something is going wrong, haven't figured out what but with running -vv you can see why a lock-file is considered out-of-date.

 INFO pixi::config: Global config not found at /Users/tdejager/.config/pixi/config.toml
 INFO pixi::config: Global config not found at /Users/tdejager/Library/Application Support/pixi/config.toml
 INFO pixi::config: Global config not found at /Users/tdejager/.pixi/config.toml
 INFO pixi::environment: verifying prefix location is unchanged, with prefix file: /private/tmp/modflow6/.pixi/envs/default/conda-meta/pixi_env_prefix
 INFO pixi::lock_file::outdated: the pypi dependencies of environment 'default' for platform win-64 are out of date because the requirement 'pandas' could not be satisfied (required by 'modflowapi')
 INFO pixi::lock_file::outdated: the pypi dependencies of environment 'default' for platform linux-64 are out of date because the requirement 'pandas' could not be satisfied (required by 'modflowapi')
 INFO pixi::lock_file::outdated: the pypi dependencies of environment 'default' for platform osx-arm64 are out of date because the requirement 'pandas' could not be satisfied (required by 'modflowapi')
 INFO pixi::lock_file::outdated: the pypi dependencies of environment 'default' for platform osx-64 are out of date because the requirement 'pandas' could not be satisfied (required by 'modflowapi')
 ...

We are investigating further.

[tdejager]

It's a bigger problem with missing packages from the new pypi mapping, we are trying to fix this now.

[ruben-arts]

The current issue is that our map is out of date with conda-forge @nichmor is working on that to fix v0.19.0. I'm going to remove the use of that map from main and make a patch release as soon as we can.

[tdejager]

We fixed the map. Locally I get the correct versions for modflow6. However, you need to regenerate the whole lock file once, sorry about that.

[Hofer-Julian]

We fixed the map. Locally I get the correct versions for modflow6. However, you need to regenerate the whole lock file once, sorry about that.

Thanks, @tdejager.
@mjr-deltares and I already wondered why updating the lockfile without deleting it first didn't work.

I've already regenerated it with pixi 0.19 and that seems to work fine. Or should I regenerate it with 0.18 to be sure?

[ruben-arts]

Would be good to see if there is now a difference between the two pixi versions, so if you could share that, that would be nice! @Hofer-Julian

[Hofer-Julian]

I ran the following:

rm pixi.lock
./pixi018 install
mv pixi.lock pixi018.lock
./pixi019 install
mv pixi.lock pixi19.lock

The lockfiles differ and that's how they look like:

modflow6_lock_files.zip

[tdejager]

Yeah, okay that's looking good, so the new lock file is only including the pypi source dependencies, which is what you want to achieve:

      - pypi: git+https://github.com/modflowpy/flopy.git@9e87acddcf0740aac491826ef86fd45ae2304dac
      - pypi: git+https://github.com/MODFLOW-USGS/modflow-devtools.git@08eff72b73da66541f7a08ddf398298210ce3758
      - pypi: git+https://github.com/MODFLOW-USGS/modflowapi.git@0c632e511fb54068d14dcdff44c572ccb6e8831f

The rest is conda so this seems to be working. Also the 0.19 one seems to include purl's which we now use for the conda to pypi comparison.

[ruben-arts]

Close as fixed