Build manylinux wheels (including aarch64) with Zuul

[ianw]

Build manylinux wheels with Zuul

This adds the Zuul playbooks and role to build manylinux wheels for
aarch64 and x86_64 (while aarch64 is the primary goal; it's good for
the overall code to keep it flexible).

It first builds an sdist from the checkout and then builds the wheels
in the appropriate containers.

Note this adds the jobs in the gate pipeline, which currently responds
to Pull Requests, and the release pipeline, which responds to pushes
to refs/tags/.* (see [1]). Note for results of jobs run against tags
you will need to find the job directly from

https://zuul.opendev.org/t/pyca/builds

because there is nowhere to report the results as such (it could be
configured to send an email).

The wheels are published to the wheelhouse/ directory in the Zuul
logs, which is also listed as an artifact on the build results page.

[1] https://review.opendev.org/748323

[opendev-zuul]

Build failed (check pipeline). For information on how to proceed, see
https://docs.opendev.org/opendev/infra-manual/latest/developers.html#automated-testing

• pyca-cryptography-build-wheel : FAILURE in 2m 47s
• pyca-cryptography-ubuntu-focal-py38-arm64 : SUCCESS in 17m 40s
• pyca-cryptography-ubuntu-bionic-py36-arm64 : SUCCESS in 17m 25s
• pyca-cryptography-ubuntu-xenial-py27-arm64 : SUCCESS in 19m 46s
• pyca-cryptography-centos-8-py36-arm64 : SUCCESS in 14m 53s

[ianw]

recheck

[opendev-zuul]

Build failed (check pipeline). For information on how to proceed, see
https://docs.opendev.org/opendev/infra-manual/latest/developers.html#automated-testing

• pyca-cryptography-build-wheel : FAILURE in 4m 44s
• pyca-cryptography-ubuntu-focal-py38-arm64 : SUCCESS in 13m 07s
• pyca-cryptography-ubuntu-bionic-py36-arm64 : SUCCESS in 13m 08s
• pyca-cryptography-ubuntu-xenial-py27-arm64 : SUCCESS in 17m 22s
• pyca-cryptography-centos-8-py36-arm64 : SUCCESS in 15m 00s

[opendev-zuul]

Build succeeded (check pipeline).

• pyca-cryptography-build-wheel : SUCCESS in 10m 33s
• pyca-cryptography-ubuntu-focal-py38-arm64 : SUCCESS in 12m 27s
• pyca-cryptography-ubuntu-bionic-py36-arm64 : SUCCESS in 12m 59s
• pyca-cryptography-ubuntu-xenial-py27-arm64 : SUCCESS in 17m 59s
• pyca-cryptography-centos-8-py36-arm64 : SUCCESS in 15m 31s

[opendev-zuul]

Build succeeded (check pipeline).

• pyca-cryptography-build-wheel : SUCCESS in 10m 09s
• pyca-cryptography-ubuntu-focal-py38-arm64 : SUCCESS in 12m 20s
• pyca-cryptography-ubuntu-bionic-py36-arm64 : SUCCESS in 12m 44s
• pyca-cryptography-ubuntu-xenial-py27-arm64 : SUCCESS in 18m 21s
• pyca-cryptography-centos-8-py36-arm64 : SUCCESS in 15m 12s

[alex]

This is temporarily blocked on the fact that the manylinux 2014 wheel building images don't seem to specify page size, so it compiles binaries that don't work on all systems.

[reaperhulk]

Is it possible to have this be a job we trigger via some API call? Our workflow for release involves pushing a tag to GitHub, uploading the sdist, then triggering the wheel building process (to ensure that the wheel builders pull the sdist and not ). As part of this we normally specify a version to build via an API call and then download the built artifacts. I see we can access artifacts from Zuul, but the trigger mechanism is unclear to me right now.

[ianw]

Is it possible to have this be a job we trigger via some API call?

Not really; you can see the pipelines Zuul is configured to run and the events that trigger that at

https://opendev.org/pyca/project-config/src/branch/master/zuul.d/pipelines.yaml

Our workflow for release involves pushing a tag to GitHub, uploading the sdist, then triggering the wheel building process (to ensure that the wheel builders pull the sdist and not ). As part of this we normally specify a version to build via an API call and then download the built artifacts. I see we can access artifacts from Zuul, but the trigger mechanism is unclear to me right now.

So the way that Zuul would handle this is we need to add a pipeline to listen for pushes matching ref/tags/<regex-to-match-the-tag-format> and then put whatever jobs we want to run in that pipeline.

For example, you can see how the main openstack release pipeline is configured at

https://opendev.org/openstack/project-config/src/branch/master/zuul.d/pipelines.yaml#L164

basically whenever openstack projects push tags, the projects all have a bunch of common jobs that live in that pipeline to create and publish source tarballs, push documentation, upload to pypi, etc.

The only trick here is that there's nowhere for Zuul to report. These jobs will appear in https://zuul.opendev.org/t/pyca/builds but you do have to know to look there.

(to ensure that the wheel builders pull the sdist and not ).

"and not the git source" I guess you mean?

This bit I need to get my head around. I mean, it's software we can do anything and figure out how to pull a sdist I'm sure :) But the Zuul job will be running on the source code at the pushed tag. I feel like we should really try to build from that for repeatability; if we're pulling an sdist made somewhere else we have windows for things to go wrong.

[ianw]

So this now makes an sdist and builds against that using bdist_wheel

Although there's no API as such, Zuul administrators can enqueue changes against a particular ref; i.e. we could enqueue this job against existing tags and build wheels for current releases if this is of interest.

As discussed above, we can also get this to watch for new tag pushes and build wheels there too.

[alex]

Ok, we now have an officla manylinux2014 aarch64 wheel building docker image: https://hub.docker.com/r/pyca/cryptography-manylinux2014_aarch64

[ianw]

@alex @reaperhulk Thanks for your interest in this

I believe this is ready for review now, and is generating good wheels from the sdist, doing basic sanity checks, and publishing them.

If this is accepted and merged, I can manually trigger these jobs to run against the 3.0 tag and thus produce aarch64 wheels which could be uploaded manually (manually injecting events like this requires Zuul command-line access, so can only be done by administrators). I believe this would be of great help to everyone struggling with slowish arm64 deployments right now.

For the future, you will see I put the jobs in the release pipeline, which means Zuul will run these jobs when tags are pushed (to be specific, when pushes to ref/tags/.* are reported). As mentioned in the change, you would then need to go to the Zuul status page and find the builds and grab the output wheels from there -- I could setup an email with the results if this is of interest.

Zuul has roles for automated uploading to pypi (https://zuul-ci.org/docs/zuul-jobs/python-roles.html#role-upload-pypi) which is used by all openstack/opendev projects doing releases, and I'd be happy to discuss this further if it is of interest. It would likely mean giving the user openstackci permission to upload.

Thanks!

[alex]

Do we need this? We currently build our x86-64 wheels elsehwere.

[ianw]

This was a deliberate choice on my behalf to make sure the playbooks/roles are (and remain) suitable for multi-arch and multi-python builds in general

[alex]

We dont' usually run our wheel builder tasks in CI... though I guess there's no reason we couldn't. WDYT @reaperhulk ?

[ianw]

I would say that this does ensure when it comes time to push a tag, you can be sure there's no surprises that you've broken something in the mean time.

[reaperhulk]

I don't have an objection to this happening provided we have the concurrency for it not to hurt our CI turnaround time.

[reaperhulk]

pep8 failure is a black release occurring since we don't pin that. #5429 will fix it.

[alex]

Ok, black is fixed on master. Should be safe to merge/rebase.

[reaperhulk]

No need to rebase, just restarted the job in this case.