Add nightly wheel upload to Anaconda Cloud

[matthewfeickert]

To move the idea of scientific-python/upload-nightly-action#29 along sooner, I've made a scikit-hep-nightly-wheels Anaconda Cloud organization (only 3 GB of storage at the moment) and made an Awkward group there.

Repurposing the workflow parts of https://github.com/matplotlib/matplotlib/blob/c2aa4ee1177452dc778813e3034c3f57d7db83d8/.github/workflows/nightlies.yml, originally added in matplotlib/matplotlib#22733, on my fork I've created a similar workflow that is able to download the results of the Packaging Tests workflow and then upload to the scikit-hep-nightly-wheels Anaconda Cloud organization using

      - name: Upload wheels to Anaconda Cloud as nightlies
        uses: scientific-python/upload-nightly-action@6e9304f7a3a5501c6f98351537493ec898728299 # 0.3.0
        with:
          artifacts_path: dist
          anaconda_nightly_upload_organization: scikit-hep-nightly-wheels
          anaconda_nightly_upload_token: ${{ secrets.ANACONDA_ORG_UPLOAD_TOKEN }}

successfully

$ docker run --rm -ti python:3.11 /bin/bash
root@0487e039366a:/# python -m venv venv && . venv/bin/activate
(venv) root@0487e039366a:/# python -m pip --quiet install --upgrade pip wheel
(venv) root@0487e039366a:/# python -m pip install --upgrade --pre  --index-url https://pypi.anaconda.org/scikit-hep-nightly-wheels/simple --extra-index-url https://pypi.org/simple awkward
Looking in indexes: https://pypi.anaconda.org/scikit-hep-nightly-wheels/simple, https://pypi.org/simple
Collecting awkward
  Downloading awkward-2.6.1-py3-none-any.whl.metadata (7.0 kB)
Collecting awkward-cpp==29 (from awkward)
  Downloading awkward_cpp-29-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.2 kB)
Collecting fsspec>=2022.11.0 (from awkward)
  Downloading fsspec-2024.2.0-py3-none-any.whl.metadata (6.8 kB)
Collecting importlib-metadata>=4.13.0 (from awkward)
  Downloading importlib_metadata-7.0.1-py3-none-any.whl.metadata (4.9 kB)
Collecting numpy>=1.18.0 (from awkward)
  Downloading numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.0/61.0 kB 1.1 MB/s eta 0:00:00
Collecting packaging (from awkward)
  Downloading packaging-23.2-py3-none-any.whl.metadata (3.2 kB)
Collecting zipp>=0.5 (from importlib-metadata>=4.13.0->awkward)
  Downloading zipp-3.17.0-py3-none-any.whl.metadata (3.7 kB)
Downloading awkward-2.6.1-py3-none-any.whl (749 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 750.0/750.0 kB 7.8 MB/s eta 0:00:00
Downloading awkward_cpp-29-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (706 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 706.6/706.6 kB 9.4 MB/s eta 0:00:00
Downloading fsspec-2024.2.0-py3-none-any.whl (170 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 170.9/170.9 kB 15.4 MB/s eta 0:00:00
Downloading importlib_metadata-7.0.1-py3-none-any.whl (23 kB)
Downloading numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.3/18.3 MB 20.4 MB/s eta 0:00:00
Downloading packaging-23.2-py3-none-any.whl (53 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.0/53.0 kB 7.1 MB/s eta 0:00:00
Downloading zipp-3.17.0-py3-none-any.whl (7.4 kB)
Installing collected packages: zipp, packaging, numpy, fsspec, importlib-metadata, awkward-cpp, awkward
Successfully installed awkward-2.6.1 awkward-cpp-29 fsspec-2024.2.0 importlib-metadata-7.0.1 numpy-1.26.4 packaging-23.2 zipp-3.17.0
(venv) root@0487e039366a:/# python -m pip list
Package            Version
------------------ --------
awkward            2.6.1
awkward-cpp        29
fsspec             2024.2.0
importlib-metadata 7.0.1
numpy              1.26.4
packaging          23.2
pip                24.0
setuptools         65.5.0
wheel              0.42.0
zipp               3.17.0
(venv) root@0487e039366a:/#

If this sounds good, I'll PR what I have and I'll need at least 1 (perferably 2) maintainers to create Anaconda Cloud (https://anaconda.org/) accounts and make new API tokens with permissions described in https://scientific-python.org/specs/spec-0004/#process-for-adding-new-projects

[matthewfeickert]

Whoops, a bit too soon on that "succesfully" as I noticed it was falling back to the default PyPI. Seems that pip is unable to find the awkward-cpp wheels even though they exist at https://anaconda.org/scikit-hep-nightly-wheels/awkward-cpp/

$ docker run --rm -ti python:3.11 /bin/bash
root@4e469e4c6b90:/# python -m venv venv && . venv/bin/activate
(venv) root@4e469e4c6b90:/# python -m pip --quiet install --upgrade pip wheel
(venv) root@4e469e4c6b90:/# python -m pip --quiet install awkward
(venv) root@4e469e4c6b90:/# python -m pip --quiet uninstall --yes awkward awkward-cpp
(venv) root@4e469e4c6b90:/# python -m pip list
Package            Version
------------------ --------
fsspec             2024.2.0
importlib-metadata 7.0.1
numpy              1.26.4
packaging          23.2
pip                24.0
setuptools         65.5.0
wheel              0.42.0
zipp               3.17.0
(venv) root@4e469e4c6b90:/# python -m pip install --upgrade --pre  --index-url https://pypi.anaconda.org/scikit-hep-nightly-wheels/simple awkward
Looking in indexes: https://pypi.anaconda.org/scikit-hep-nightly-wheels/simple
Collecting awkward
  Downloading https://pypi.anaconda.org/scikit-hep-nightly-wheels/simple/awkward/2.6.1/awkward-2.6.1-py3-none-any.whl (763 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 763.5/763.5 kB 1.8 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of awkward to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement awkward-cpp==29 (from awkward) (from versions: none)
ERROR: No matching distribution found for awkward-cpp==29
(venv) root@4e469e4c6b90:/# python -m pip index --index-url https://pypi.anaconda.org/scikit-hep-nightly-wheels/simple versions awkward
WARNING: pip index is currently an experimental command. It may be removed/changed in a future release without prior warning.
awkward (2.6.1)
Available versions: 2.6.1
(venv) root@4e469e4c6b90:/# python -m pip index --index-url https://pypi.anaconda.org/scikit-hep-nightly-wheels/simple versions awkward-cpp
WARNING: pip index is currently an experimental command. It may be removed/changed in a future release without prior warning.
WARNING: Cache entry deserialization failed, entry ignored
ERROR: No matching distribution found for awkward-cpp
(venv) root@4e469e4c6b90:/#

edit: Ah, yeah, because I was using Python 3.11 and apparently only a subset of the wheels are getting built in .github/workflows/packaging-test.yml.

[matthewfeickert]

Okay yeah, everything is good with Python 3.12.

It turns out that the way that pip install --pre works and that the naming conventions of the wheels that are generated in the https://github.com/scikit-hep/awkward/blob/dd4753b97949e4098aa6008a189624ba7a72fda6/.github/workflows/packaging-test.yml workflow that from a clean virtual environment the standard approach of setting the nightly index as --index-url and normal PyPI as --extra-index-url

python -m pip install --upgrade --pre --index-url https://pypi.anaconda.org/scikit-hep-nightly-wheels/simple --extra-index-url https://pypi.org/simple awkward

doesn't work as the released version on PyPI will be seen as the "later" one and go with that.

To work around this, if we install awkward first from PyPI and then immediatley uninstall awkward and awkward-cpp so that we're just left with their dependencies, then we can install from the nightly package index directly

$ docker run --rm -ti python:3.12 /bin/bash
root@14d404c25c0d:/# python -m venv venv && . venv/bin/activate
(venv) root@14d404c25c0d:/# python -m pip --quiet install --upgrade pip wheel
(venv) root@14d404c25c0d:/# python -m pip --quiet install --upgrade awkward && python -m pip uninstall -y awkward awkward-cpp && python -m pip list
Found existing installation: awkward 2.6.1
Uninstalling awkward-2.6.1:
  Successfully uninstalled awkward-2.6.1
Found existing installation: awkward-cpp 29
Uninstalling awkward-cpp-29:
  Successfully uninstalled awkward-cpp-29
Package   Version
--------- --------
fsspec    2024.2.0
numpy     1.26.4
packaging 23.2
pip       24.0
wheel     0.42.0
(venv) root@14d404c25c0d:/# python -m pip install --upgrade --pre --index-url https://pypi.anaconda.org/scikit-hep-nightly-wheels/simple awkward
Looking in indexes: https://pypi.anaconda.org/scikit-hep-nightly-wheels/simple
Collecting awkward
  Downloading https://pypi.anaconda.org/scikit-hep-nightly-wheels/simple/awkward/2.6.1/awkward-2.6.1-py3-none-any.whl (763 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 763.5/763.5 kB 2.2 MB/s eta 0:00:00
Collecting awkward-cpp==29 (from awkward)
  Downloading https://pypi.anaconda.org/scikit-hep-nightly-wheels/simple/awkward-cpp/29/awkward_cpp-29-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (707 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 707.7/707.7 kB 1.1 MB/s eta 0:00:00
Requirement already satisfied: fsspec>=2022.11.0 in /venv/lib/python3.12/site-packages (from awkward) (2024.2.0)
Requirement already satisfied: numpy>=1.18.0 in /venv/lib/python3.12/site-packages (from awkward) (1.26.4)
Requirement already satisfied: packaging in /venv/lib/python3.12/site-packages (from awkward) (23.2)
Installing collected packages: awkward-cpp, awkward
Successfully installed awkward-2.6.1 awkward-cpp-29
(venv) root@14d404c25c0d:/# python -m pip list
Package     Version
----------- --------
awkward     2.6.1
awkward-cpp 29
fsspec      2024.2.0
numpy       1.26.4
packaging   23.2
pip         24.0
wheel       0.42.0
(venv) root@14d404c25c0d:/#