exclude openblas-libs from the cleanup job

[mattip]

The files at https://anaconda.org/scientific-python-nightly-wheels/openblas-libs/files are not meant to be cleaned up automatically. We generate those tarballs (which are not conda packages) for long-term use by SciPy and NumPy. Is there a way to exclude them from https://github.com/scientific-python/upload-nightly-action/blob/main/.github/workflows/remove-wheels.yml ? If not we should go back to keeping them at https://anaconda.org/multibuild-wheels-staging/openblas-libs/

CC @honno, @rgommers

[mattip]

It seems there is a retention policy for packages that the remove-wheels github action implements. Perhaps the policy could be added to SPEC-04 and the org description could add some detail or point to the SPEC.

[matthewfeickert]

Is there a way to exclude them from https://github.com/scientific-python/upload-nightly-action/blob/main/.github/workflows/remove-wheels.yml ?

The technical answer is easy. Just after

upload-nightly-action/.github/workflows/remove-wheels.yml

Lines 52 to 56 in 2b318b6

	- name: Query package index for packages
	run: |
	anaconda show "${ANACONDA_USER}" &> >(grep "${ANACONDA_USER}/") | \
	awk '{print $1}' | \
	sed 's|.*/||g' > package-names.txt

insert

sed -i '/openblas-libs/d' package-names.txt

as part of the "Query package index for packages" step.

It seems there is a retention policy for packages that the remove-wheels github action implements.

The question of if packages that aren't meant to be removed and aren't actually nightlies should be housed here or not seems related to Issue #30.

[matthewfeickert]

@mattip @honno Can you confirm that all the versions that are current uploaded (v0.3.23-246-g3d31191b, v0.3.23-293-gc2f4bdbb, and HEAD) are needed currently needed?

@bsipocz has already assigned herself to this Issue (:+1:), but I just wanted to get this clarified as if v0.3.23-246-g3d31191b need to be guarded (some of the uploads for that version are 28 days old today) this should be done with a 1 liner PR I gave above quickly to avoid any issues while the discussion of where non-nightly things live is had.

[mattip]

The opnblas-libs are more like PyPI artifacts, in that various versions of NumPy and Scipy will ask for various versions of these artifacts. The versions are specified in a script, here for NumPy and here for Scipy (which is still using the old site for the non-nightly version). Version v0.3.23-246-g3d31191b is the one used by the previous version of the script for NumPy/main, and as far as I can tell is not used by any active branches or release tags.

This discussion is one more reason to turn the openblas tarballs into wheels, see MacPython/openblas-libs#86.

[matthewfeickert]

Version v0.3.23-246-g3d31191b is the one used by the previous version of the script for NumPy/main, and as far as I can tell is not used by any active branches or release tags.

Thanks for verifying @mattip. As v0.3.23-293-gc2f4bdbb was uploaded 14 days ago then I think we can leave things as is as for the time being and people can discuss what's the best way to move forward (I'll defer to others here and probably the discussions happening on other Issues now). If the NumPy and SciPy teams would strongly prefer to have things guarded now though I have a branch with this guard on it ready to push.

[mattip]

My request still stands: please exclude openblas-libs from the cleanup job, regardless of ongoing conversations about whether the use of this site is a good fit for openblas-libs. I may be wrong about the need for the artifact about to be deleted, and it would be a shame to break somebody's build.

[matthewfeickert]

👋 @mattip The openblas-libs files are currently using 4.6 GB out of the org's (current) total 20 GB and 3.4 GB worth of those files have never been downloaded.

This is currently fine, but as you asked that all automated removal of the files be excluded for these files, what is the openblas-libs's team's plan for curation of the files?

[mattip]

Once numpy/numpy#25505 goes in, and a similar PR can be made to SciPy, we will stop uploading tarballs and only upload wheels (scipy-openblas64 and scipy-openblas32).

[matthewfeickert]

@mattip very cool! Congrats in advance and well done to you and the rest of the team on this work.

edit: Though does that mean that the wheel can be treated as normal and no longer need to be reserved artifacts that can't be cleaned up?

[mattip]

Though does that mean that the wheel can be treated as normal a

The wheel or the tarballs? The tarballs (at openblas-libs) will still be used by older builds of maintenance versions and by third-party projects for a while. I don't think the wheels (at scipy-openblas64 and scipy-openblas32) were ever reserved artifacts, only the tarballs, right?

[matthewfeickert]

I don't think the wheels (at scipy-openblas64 and scipy-openblas32) were ever reserved artifacts, only the tarballs, right?

Right.

The tarballs (at openblas-libs) will still be used by older builds of maintenance versions and by third-party projects for a while.

Okay, I think this answers my question, thanks. There can be some natural phase out period where at some future point (would you project 1 year as reasonable?) the tarballs can be removed, and then just the wheels can remain and those can get cycled at the same rates as the rest of the projects. 👍

[matthewfeickert]

Congratulations @mattip to you and the rest of the team on getting numpy/numpy#25505 in! 🎉 That seems like a heroic amount of work that has been done, but seems exciting.

For those of us who haven't been fully tracking this work (i.e. me) and its ramifications for NumPy and SciPy wheels, can you or @rgommers give just a bullet list, to compliment #34 (comment), of what this means moving forward for what will get uploaded to the scientific-python-nightly-wheels Anaconda org, and what Issues or PRs interested parites should be watching?

I think the answer to the first part, given #34 (comment), is that we should soon expect there to be no new openblas-libs uploads and that the only future uploads will be scipy-openblas64 and scipy-openblas32 wheels. Though what is the projected timeline for that? (now? in a few weeks? next release?)

[matthewfeickert]

Issues/PRs to watch (running list):

• MAINT: Remove openblas_support.py scipy/scipy#20074