Cannot install awkward from HEAD: awkward-cpp==3 not found

[alexander-held]

Version of Awkward Array

1e475cf (current HEAD)

Description and code to reproduce

Hi, installing the HEAD of awkward via pip currently fails while trying to find awkward-cpp==3:

$ pip install git+https://github.com/scikit-hep/awkward.git
[...]
ERROR: Could not find a version that satisfies the requirement awkward-cpp==3 (from awkward) (from versions: 0.11.0rc7.linux-x86_64, 0.11.0rc8.linux-x86_64, 0.10.4, 0.11.1, 0.12.0rc1, 0.12.0rc2, 0.12.0, 0.12.1, 0.12.2, 0.12.3, 0.12.4, 0.12.5, 0.12.6, 0.12.7, 0.12.8, 0.12.9, 0.12.10, 0.12.11, 0.12.12, 0.12.13, 0.12.14, 0.12.15, 0.12.16, 0.12.17, 0.12.18, 0.12.19, 0.12.20, 0.12.21, 0.12.22, 0.13.0, 0.14.0, 1, 2)
ERROR: No matching distribution found for awkward-cpp==3

From a brief look at the history it seems like this may be due to 307fa10 raising the version number to 3.

I ran the above in a python:3.10-slim container to reproduce, but presumably this is not specifically an environment issue.

[agoose77]

We bump our awkward-cpp version/dependency whenever the C++ package changes e.g. due to new kernels. The new ak.drop_none() will do this, so we've bumped the version in anticipation. I wonder whether we should do that afterwards, but either strategy leads to pip install git+... not working.

The problem here is really that we have a monorepo, so you realistically need to clone the repo in order to build the packages. Related to this is our two-phase build; there's a prepare stage that produces the files required for an sdist to be buildable (e.g. generating test files, copying shared headers).

I think practically speaking one has to do this in a two phase build, e.g.

pushd $(mktemp -d)

git clone --depth=1 --recurse-submodules --shallow-submodules https://github.com/scikit-hep/awkward.git
cd awkward

pipx run nox -s prepare

mkdir wheels
pipx run  --python $(which python3) build -w -o wheels awkward-cpp
pipx run  --python $(which python3) build -w -o wheels .

However, I'm curious as to why the built awkward-cpp has the manylinux_2-35 tag for me: this should produce a native Linux wheel.

[jpivarski]

We won't be able to support

pip install git+https://github.com/scikit-hep/awkward.git

installations anymore, at least not in a way that will always work. (We never claimed to support this before, but I guess it would have worked if you had all of the build tools installed.)

We still need to figure out a policy for when PRs that touch C++ get merged and when the awkward-cpp version number changes. What happened here is that I updated the version number before the two PRs that need it (#1904 and #2001) have been merged because the script that protects us against forgetting to update the version number is temporarily disabled and I wanted to ensure that we don't forget. With protection in place, we could have updated the version number after the PRs that need it, or we could have a policy that if a PR touches C++, it has to update the awkward-cpp version number in that PR, so that the change happens atomically. (Then what if two PRs both update it and there's no release in between? Like #1904 and #2001? The version number increase ought to be relative to the last released awkward-cpp...)

But anyway, ANY decision we make on coordinating C++ changes with awkward-cpp version number updates will not fix pip install git+https://.... There will always be a window of time between (a) when a C++ change merges into main and/or the awkward-cpp version number in main changes and (b) when an awkward-cpp version is released to PyPI. During time between (a) and (b), pip-installing through git will produce an awkward package that either knows it needs an awkward-cpp that isn't in PyPI (what you reported) or doesn't know that it needs an awkward-cpp that isn't in PyPI (that is, it will crash at runtime). Considering that error messages are better than incorrect behavior, I suppose we always want to update the awkward-cpp version number at least as early as changes to C++ (partially answering the policy question of the previous paragraph).

Since pip-installing from git must fail in time intervals between (a) and (b), we shouldn't claim that it's a supported installation mechanism in general. Perhaps C++ updates are rare, but we don't want people to rely on something that we know won't work sometimes. The actual build and upload of an awkward-cpp package takes an hour, and invoking it is a manual process with only one Release Manager (me), so the time interval between (a) and (b) can be a few days.

Beyond the coordination issue, I think pip installing from git also won't work because some of the files need to be built with nox, as @agoose77 described. Some of these generated files are part of the awkward package (not just awkward-cpp), so even if the correct awkward-cpp is already in PyPI, it won't work. The motivating reason for that was so that the pyproject.toml can be declarative (not have it run the Python scripts that generate the missing files), but since we can't rely on pip installing from git at some times, I don't think breaking the declarativeness of pyproject.toml is justified.

So as an action item for this issue, we need a sentence in README.md, after the pip install instructions, to say that pip install from the git repo will not work. I tried searching for other places where installation is described. There's this one in the docs: docs/getting-started/index.md but it's a very short help box; too brief to include any additional clarifications. (Besides, it would be nice if this information is only in one place.) Someone who's trying to pip install from git wants the Installation for Developers section, just below it.

Actually, I'll go add this now.

[agoose77]

Just to add on our conversation earlier - we can have scripts running in pyproject.toml through plugins, but it adds complexity that we don't have a strong justification for. In particular, our sdists are simple because they don't bundle any dev scripts - everything is generated for them. This is a nice distinction that is prefer to keep, hence the 'declarative' point.

I.e, at the 'package' level, awkward and awkward-cpp are totally distinct.

[jpivarski]

That's what I meant in my

Beyond the coordination issue, ...

paragraph. If that were the only thing preventing pip install through git, we'd want to reconsider it: weighing developer experience versus user experience. (One is not automatically more important than the other, but we'd have to think about the relative value.)

However, since it must be the case that installation through git will be broken in time intervals between (a) and (b), it's obvious: we're not going to recommend an installation procedure that necessarily must be broken some of the time. So we don't have to consider sacrificing the declarativeness of pyproject.toml.

[agoose77]

Oh absolutely! I just wanted to clarify the technical possibility of running scripts in our build chain! :)