Update CI to publish binary releases on git version tag #27

almarklein · 2020-05-19T19:40:09Z

This adds jobs to the Github action CI workflow, that build binaries suited for distribution. On commits that are tagged with v..., a matching release is made and the binaries (packed into zipfiles) are uploaded as assets.

I basically took the Azure pipeline code from wgpu-bin, translated it to Github Actions, and weaved in into the existing CI. I deliberately made a new release-build job, so we can keep these builds separate from the test-build jobs.

I tested this on my own fork (Github Actions Just Work on forks!), I pushed a tag, and some minutes later the releases appeared.

So to publish a release, we must bump the version in cargo.toml, and then git tag v.. and git push that tag. I considered detecting the version bump and creating a tag automatically, but IMO that feels like too much automation. Open for discussion though.

We have quite a lot of jobs now, some kind of duplicate. We could consider removing (some of) the stable test builds.

almarklein · 2020-05-19T19:42:24Z

@kvark would you mind if I squash-merged this when we're ready? If I read it correctly, I can add co-authors then, and I'd like to attribute @Korijn, who authored wgpu-bin.

almarklein · 2020-05-19T19:45:23Z

Two of the new builds pull a Docker image from quay.io, (by RedHat) which has been having issues all day. (had to restart CI a few times just now to make it work) I assume that should normally be pretty stable :)

almarklein · 2020-05-19T19:51:39Z

.github/workflows/ci.yml

-          Ubuntu Nightly,
-          Windows Stable,
-          Windows Nightly
-        ]


Removed this, as it's not needed, I believe.

Turns out this was here for bors, see bors.toml :)

Does it though? The jobs still have the same names. I don't think Bors checks this file or anything, it just listens for success/fail events with matching names, so I think it's fine. Unless I'm missing something.

kvark

Thank you, this looks very promising (and sophisticated:) )

kvark · 2020-05-20T15:38:44Z

.github/workflows/ci.yml

-    name: Publish release artifacts
-    runs-on: ubuntu-18.04
+  # Builds for binary releases.
+  # Also build on non-tag commits; it also functions as a test.


Do we have an idea of how much slower this would make our general CI? I.e. does using an image affect the build times?

Korijn · 2020-05-20T17:25:50Z

In my experience, the rust build still largely dominates the time. The docker image is pretty small so its pulled in no time.

Running it within a docker image on Linux shouldn't effect the build time at all.. it's still running as a process on the host (no virtualization or anything)

almarklein · 2020-05-20T19:31:17Z

Looking at the reported times for each job, they are indeed very much comparable. The Windows builds take the longest, but they take about as long as the Windows builds that were already done.

There are more jobs now, so potentially it can take more time if there's a queue for the pool of runners.

kvark

Alright, thank you!

kvark · 2020-05-20T19:33:09Z

@grovesNL would you be interested to look at this as well? I think it would be useful. Otherwise, we'll proceed.

Korijn · 2020-05-20T20:25:25Z

.github/workflows/ci.yml

+      uses: actions/upload-artifact@v2
+      with:
+        path: commit-sha
+        name: dist


Unlike wgpu-bin where we have to clone the source repo prior to building, here you can just create the commit-sha file in the final publish job and avoid the prepare job completely. It will allow queueing the subsequent jobs earlier

I'd assumed the build needed it, but make package already does that. We can thus omit this step completely, because the GH release page already reports the commit sha :)

Korijn · 2020-05-20T20:26:46Z

.github/workflows/ci.yml

+        IMAGE: ${{ matrix.IMAGE }}
+        RUST_TOOLCHAIN: ${{ matrix.RUST_TOOLCHAIN }}
+      run: |
+          CID=$(docker create -t -w /tmp/wgpu-native -v $PWD:/tmp/src:ro quay.io/pypa/$IMAGE bash -c "\


Don't you need to preface all of the bash scripts with set -ex to make sure we don't steamroll errors? Or is that by default on github actions?

The default is set -e (-like behavior), which can be turned off with fail-fast: false

Ok, that's fine then

grovesNL

Thank you! This looks great to me.

The Docker idea is interesting and makes we wonder whether we should be doing something similar in gfx-portability too.

grovesNL · 2020-05-21T02:08:56Z

.github/workflows/ci.yml

-    name: Publish release artifacts
-    runs-on: ubuntu-18.04
+  # Builds for binary releases.
+  # Also build on non-tag commits; it also functions as a test.


I'd slightly prefer to only bother building tagged commits to avoid the potential for CI backlog, but we could start with this and see how it goes

kvark · 2020-05-21T03:47:17Z

Sounds like we are good to merge this.
There are concerns voiced by @Korijn though that would need to be resolved.
bors delegate=Korjin

bors · 2020-05-21T03:47:18Z

✌️ KORJin can now approve this pull request. To approve and merge a pull request, simply reply with bors r+. More detailed instructions are available here.

grovesNL · 2020-05-21T03:55:08Z

(username typo)
bors delegate=Korijn

bors · 2020-05-21T03:55:09Z

🔒 Permission denied

Existing reviewers: click here to make grovesNL a reviewer

grovesNL · 2020-05-21T03:56:06Z

cc @kvark ^

kvark · 2020-05-21T03:56:47Z

bah
bors delegate=Korijn

bors · 2020-05-21T03:56:48Z

✌️ Korijn can now approve this pull request. To approve and merge a pull request, simply reply with bors r+. More detailed instructions are available here.

kvark · 2020-05-21T03:58:47Z

@grovesNL sorry about that! fixed now, you should have full control here.

Korijn · 2020-05-21T06:47:04Z

No worries, it happens literally all the time! :)

Korijn · 2020-05-21T06:52:00Z

@grovesNL just to make sure we are all clear on why we're using that particular docker container:

it helps create highly portable binaries for linux for reasons better explained here: https://www.python.org/dev/peps/pep-0513/
the python package index only accepts binary packages that adhere to the linked PEP513 standard, so it was a requirement for wgpu-py

Korijn · 2020-05-21T14:11:44Z

LGTM. @almarklein might want to give it one last full run at your fork?

almarklein · 2020-05-21T16:04:42Z

@Korijn That was a good call. Some fixes that I had earlier made to the release job had disappeared as I prepared this new branch. Anyway:

λ git tag v0.5.test5
λ git push origin v0.5.test5  # (origin being my fork)

-> https://github.com/almarklein/wgpu-native/releases/tag/v0.5.test5

Korijn · 2020-05-21T16:56:27Z

bors r+

kvark · 2020-05-21T19:56:00Z

Strange, let me try
bors r=Korijn

kvark · 2020-05-21T19:56:49Z

It shows some internal server error :/
Do we need to update bors.toml?

Korijn · 2020-05-21T20:22:48Z

@almarklein any idea why build status is still pending?

kvark · 2020-05-21T22:28:19Z

We'll need to merge this manually. That's ok. But let's also update "bors.toml" with the new task names.

kvark · 2020-05-22T15:23:22Z

Also somewhat minor - there is a separate section in Readme describing the binary releases, which we should update according to this PR.

kvark · 2020-05-22T21:19:10Z

Are we all set here?

almarklein · 2020-05-22T21:32:23Z

I think so. I made the release-builds only build for tags. The docker registry that the image is to be pulled from is having issues again, so I'm taking this as a sign that it's a liability.

Other than that, what up with that failing MacOS build suddenly?

Korijn · 2020-05-23T09:10:43Z

I think so. I made the release-builds only build for tags. The docker registry that the image is to be pulled from is having issues again, so I'm taking this as a sign that it's a liability.

quay.io is a very very common docker image registry, it's not normal for it to have issues like this.

Other than that, what up with that failing MacOS build suddenly?

It fails when compiling dependency ron, probably an upstream issue?

   Compiling ron v0.5.1
thread 'rustc' panicked at 'index out of bounds: the len is 0 but the index is 0', src/librustc_mir_build/hair/pattern/_match.rs:1184:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.45.0-nightly (9310e3bd4 2020-05-21) running on x86_64-apple-darwin

note: compiler flags: -C embed-bitcode=no -C debuginfo=2 --crate-type lib

note: some of the compiler flags provided by cargo are hidden

error: could not compile `gfx-backend-metal`.

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: build failed
##[error]Process completed with exit code 101.

kvark · 2020-05-23T15:03:40Z

This is coming from gfx-backend-metal (and not RON), filed as rust-lang/rust#72467
This PR needs to be merged manually, so it's not a blocker for you.

update CI to publish binary releases on git version tag

0e9b5c8

monocodus bot approved these changes May 19, 2020

View reviewed changes

almarklein commented May 19, 2020

View reviewed changes

fix type and add comment

4b9e848

monocodus bot approved these changes May 19, 2020

View reviewed changes

kvark reviewed May 20, 2020

View reviewed changes

kvark approved these changes May 20, 2020

View reviewed changes

Korijn reviewed May 20, 2020

View reviewed changes

grovesNL approved these changes May 21, 2020

View reviewed changes

almarklein added 2 commits May 21, 2020 15:07

remove prepare job

a0dce9f

more clear job names

ed5ee3b

monocodus bot approved these changes May 21, 2020

View reviewed changes

Create commit-sha file in final job

10f5f05

monocodus bot approved these changes May 21, 2020

View reviewed changes

Somehow this old (wrong) code had got in

28dbffc

monocodus bot approved these changes May 21, 2020

View reviewed changes

almarklein added 2 commits May 21, 2020 17:37

GH tells me we should update the macOS image

7474790

More code that got lost somewhere in the process

e39b44d

monocodus bot approved these changes May 21, 2020

View reviewed changes

almarklein added 3 commits May 22, 2020 22:29

align names in ci workflow and bors

0c001bd

adjust readme

60f5080

add some info to the release body

e28df46

monocodus bot approved these changes May 22, 2020

View reviewed changes

only run release builds on tag

63999b4

monocodus bot approved these changes May 22, 2020

View reviewed changes

remove comment that no longer applies

464eb8b

monocodus bot approved these changes May 22, 2020

View reviewed changes

typo

aaffae5

monocodus bot approved these changes May 22, 2020

View reviewed changes

kvark merged commit f22fa51 into gfx-rs:master May 23, 2020

almarklein deleted the cd2 branch May 23, 2020 21:19

Korijn mentioned this pull request May 26, 2020

Archive this repo when upstream releases Korijn/wgpu-bin#10

Closed

Update CI to publish binary releases on git version tag #27

Update CI to publish binary releases on git version tag #27

Conversation

almarklein commented May 19, 2020

almarklein commented May 19, 2020

almarklein commented May 19, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvark left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Korijn commented May 20, 2020 • edited Loading

almarklein commented May 20, 2020

kvark left a comment

Choose a reason for hiding this comment

kvark commented May 20, 2020

Korijn May 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Korijn May 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grovesNL left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kvark commented May 21, 2020

bors bot commented May 21, 2020

grovesNL commented May 21, 2020

bors bot commented May 21, 2020

grovesNL commented May 21, 2020

kvark commented May 21, 2020

bors bot commented May 21, 2020

kvark commented May 21, 2020

Korijn commented May 21, 2020

Korijn commented May 21, 2020

Korijn commented May 21, 2020

almarklein commented May 21, 2020

Korijn commented May 21, 2020

kvark commented May 21, 2020

kvark commented May 21, 2020

Korijn commented May 21, 2020

kvark commented May 21, 2020

kvark commented May 22, 2020

kvark commented May 22, 2020

almarklein commented May 22, 2020

Korijn commented May 23, 2020

kvark commented May 23, 2020

almarklein commented May 19, 2020 •

edited

Loading

Korijn commented May 20, 2020 •

edited

Loading

Korijn May 20, 2020 •

edited

Loading

Korijn May 20, 2020 •

edited

Loading