Cross compile for Linux, MacOS and Windows on CI #2665

owenthereal · 2023-07-07T07:41:18Z

This PR includes the following changes:

Cross compile for Linux, MacOS and Windows on CI and upload corresponding artifacts to a release (example)
Remove ubuntu-20.04 build because
- A static bin couldn't be built with the gcc toolchain
- The aarch64 clang build failed for weird reasons
Remove gcc compile target on MacOS because it symlinks gcc to clang
Remove compiler variant for Windows because only gcc is supported
Speed up release job by not waiting on docker job because it doesn't require anything from the docker job

wader · 2023-07-07T07:59:37Z

.github/workflows/ci.yml

@@ -48,10 +49,15 @@ jobs:
            --disable-maintainer-mode \
            --disable-valgrind \
            --with-oniguruma=builtin \
-            ${{ matrix.configure_flag }}
+            --host=${{ matrix.arch }}-linux-gnu \


I started looking into doing cross builds yesterday but ended up reading old autoconf documentation about host, target and build and got irritated 😄 what a mess. Also a bit confused why jq's configure script only has --build and --host... and --host is what i would call "target"? :) it's generated by autoconf 2.71 which is quite modern.

Anyways thanks doing this!

@wader https://github.com/jqlang/jq/wiki/Cross-compilation

I added cross-compilation for Windows as well. However, I noticed a few things:

Tests with the mingw-w64-x86_64 build failed for segmentation fault: (Maybe the same issue as the clang issue). I fallback to using msys gcc for x86_64 build

For Windows i386 build, I needed to add the __MINGW32__ guard to include <pthread.h> instead of the using pthread_* functions defined in jv.c & jv_thread.h due to "multiple definition" of the same functions (Ref).

wader · 2023-07-07T08:04:32Z

.github/workflows/ci.yml

          make
-          strip jq
+          ${ARCH}-linux-gnu-strip jq
+          file ./jq


Accidentally left in? i think it could make sense to keep

It's debugging, but it doesn't hurt.

The file line is to ensure the cross-compiled binary is what we expect. I guess it could be removed but it doesn't hurt to have that visibility.

.github/workflows/ci.yml

wader · 2023-07-07T08:17:52Z

A reason to wait for the docker job it make sure we notice if it fails

wader · 2023-07-07T08:22:21Z

Should we try to not to mix arm64/aarch64 and x86_64/amd64? just use one?

wader · 2023-07-07T08:22:27Z

ARM build for windows could be interesting but maybe for later?

wader · 2023-07-07T08:31:23Z

Should we try to not to mix arm64/aarch64 and x86_64/amd64? just use one?

Sorry now i noticed this is already done in release step 👍

.github/workflows/ci.yml

itchyny · 2023-07-07T12:49:40Z

.github/workflows/ci.yml

@@ -258,7 +269,7 @@ jobs:
    runs-on: ubuntu-latest
    permissions:
      contents: write
-    needs: [linux, macos, windows, dist, docker]
+    needs: [linux, macos, windows, dist]


We should abort releasing if docker build/publish fails. Even if this change releases the artifact quickly, this isn't good.

To clarify, the release job creates a draft release instead of releasing it. However, I understand you want to wait for it before creating the draft release. I put it back, provided that you will improve the docker job speed by adopting @userdocs's suggestions.

.github/workflows/ci.yml

userdocs · 2023-07-07T14:44:35Z

So I was looking at this closed pr #2652 and it seems apparent the obvious issue was that you are building via emulation, instead of cross compiling and then adding the prebuilt binary to the respective target arch, which would dramatically decrease the time to completion for that job. There is no real good reason to emulate the entire process and not on Github, you'd need to use something like https://buildjet.com/for-github-actions to make it faster.

Debian has these toolchains available https://packages.debian.org/source/bookworm/build-essential and when you use that to cross build you can do something like this perhaps?

Note: Using a debian docker (no need for ubuntu specifically here as they get these from Debian anyway and we are building statically)

https://github.com/userdocs/jq-cross-debian/actions/runs/5487472986 to publish binaries as release assets https://github.com/userdocs/jq-cross-debian/releases/tag/master-6944d81

And then just copy those to a docker image which will provide the same outcome at in a fraction of the time.

So now you have a full suite of cross built target arch static binaries as release assets people can download and then a very simple docker build process that takes simple copies the binary from the release asset or artifacts of the job.

In my example CI you can see the file outputs per build

https://github.com/userdocs/jq-cross-debian/actions/runs/5487472986/jobs/9998971987

For example,

amd64

jq-amd64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=6e8d99f61ae73fd3509778c6ff5a18076251c1a3, for GNU/Linux 3.2.0, stripped

s390x

jq-s390x: ELF 64-bit MSB executable, IBM S/390, version 1 (SYSV), statically linked, BuildID[sha1]=32615f38f2b33413a77bcd70432d87bbb87517c4, for GNU/Linux 3.2.0, stripped

Personally i'd do musl static but since you are already using a debian based solution and the binary appears to be be fullly static with no glibc dependency using crossbuild-essential works well, i think.

itchyny · 2023-07-08T01:21:16Z

@userdocs Thank you. I'll improve the docker job to use the pre-built executables.

EDIT: I have fixed the docker job and it runs within 3 minutes to build the multi-platform image. What a big improvement.

itchyny · 2023-07-08T02:08:41Z

.github/workflows/ci.yml

+          cp artifacts/jq-linux-ubuntu-22.04-gcc-x86_64/jq release/jq-linux-amd64
+          cp artifacts/jq-linux-ubuntu-22.04-gcc-aarch64/jq release/jq-linux-arm64
+          cp artifacts/jq-macos-macos-13-clang-x86_64/jq release/jq-macos-amd64
+          cp artifacts/jq-macos-macos-13-clang-arm64/jq release/jq-macos-arm64


We can mv instead of cp here.

Only `gcc` is supported well for msys2.

The CI release job doesn't need anything from the Docker job

MinGW includes pthread related functions. Ref: https://github.com/owenthereal/jq/actions/runs/5496335434/jobs/10016293713#step:4:1237

Tests failed for mingw64 build

`LDFLAGS=-s` already strip symbols

owenthereal · 2023-07-09T07:08:51Z

Thanks @userdocs for the awesome example of cross-compiling on Linux with crossbuild-essential! I adopted your approach for the Linux build in this PR. I didn't change the docker job yet, considering @itchyny is going to. I imagine the docker job will copy the Linux artifacts from the linux job to a docker image.

Besides, I added cross-compliation for Windows. Now we have complete coverage of builds for most platforms 😄. The release job is also updated to upload all the artifacts. Here is an example.

Last but not least, we wouldn't need to explicitly strip and LDFLAGS=-s will also strip symbols.

This PR is ready to be reviewed again.

userdocs · 2023-07-09T09:47:04Z

@owenthereal glad it helped. I had used LDFLAGS since the configure appears to look for a non existent strip binaries and never find it, perhaps meaning the configure needs to be overhauled at some point. With the linker flags, the gcc doing the cross compiling will strip it using strip binary it comes with, so made sense as way around that.

and also for @itchyny, As for docker, after my post, I attempted to add a matrix multi platform (which would further increase speed) job but it's not quite working. Following this https://docs.docker.com/build/ci/github-actions/multi-platform/ it seems it need to merge the manifests in a following job.

I think a combination of the prebuilt binaries + matrixed docker multi platform job would be the fastest it could be.

But the matrix part of the docker/buildx job seems to be adding complexity and not sure what the actual benefit of that is yet.

userdocs · 2023-07-09T09:59:28Z

@owenthereal also, since you are using ubuntu they provide a riscv64 crossbuild that Debian does not.

https://packages.ubuntu.com/jammy/crossbuild-essential-riscv64

Since i used Debian i did not provide it but if you add it for ubuntu you used their full suite.

https://packages.ubuntu.com/search?suite=jammy&searchon=names&keywords=crossbuild-essential

      arch:
          [
            riscv64,
          ]
        include:
          - arch: riscv64
            CC: "riscv64-linux-gnu"

.github/workflows/ci.yml

owenthereal · 2023-07-09T16:17:19Z

@userdocs:

also, since you are using ubuntu they provide a riscv64 crossbuild that Debian does not.

Added riscv64 to the list. And build passed 😄

itchyny

Thank you!

wader · 2023-07-09T23:35:24Z

Close #2386?

owenthereal changed the title ~~Cross compile arm64 on Linux & MacOS~~ Cross compile arm64 for Linux & MacOS on CI Jul 7, 2023

wader reviewed Jul 7, 2023

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

wader reviewed Jul 7, 2023

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

itchyny added this to the 1.7 release milestone Jul 7, 2023

itchyny added the release/packaging label Jul 7, 2023

itchyny requested changes Jul 7, 2023

View reviewed changes

itchyny reviewed Jul 8, 2023

View reviewed changes

owenthereal added 15 commits July 8, 2023 23:39

Support arm64 build for MacOS

42eb26e

Support aarch64 build for Linux

97433c3

Properly cross compile for MacOS

7e84859

Remove compiler matrix from Windows

15c745b

Only `gcc` is supported well for msys2.

Make release job not wait for docker job

8499241

The CI release job doesn't need anything from the Docker job

Cross compile to more Linux platforms

66ab902

Simplify MacOS target

0b31121

Fix cp the right bin for release job

2d8cc4f

Add missing Linux strip

1197756

Cross compile for Windows

5f3dc17

Don't define pthread related functions for MinGW

0daf867

MinGW includes pthread related functions. Ref: https://github.com/owenthereal/jq/actions/runs/5496335434/jobs/10016293713#step:4:1237

Upload Windows artifacts

d4709a5

Use msys gcc for Win64 build

c39a75e

Tests failed for mingw64 build

Rename upload artifacts

362c2b3

Remove explicit strip

e0612a5

`LDFLAGS=-s` already strip symbols

owenthereal added 3 commits July 8, 2023 23:39

Revert that release job waits for docker job

b59eea1

make -j$(proc) as much as possible

346ff00

Remove unused log

43892ef

owenthereal force-pushed the master branch from f7c42c2 to 43892ef Compare July 9, 2023 06:39

owenthereal changed the title ~~Cross compile arm64 for Linux & MacOS on CI~~ Cross compile arm64 for Linux, MacOS and Windows on CI Jul 9, 2023

itchyny reviewed Jul 9, 2023

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

owenthereal added 2 commits July 9, 2023 08:57

Fix quoting of *

8b486ab

Add riscv64 build for Linux

4471bf9

owenthereal changed the title ~~Cross compile arm64 for Linux, MacOS and Windows on CI~~ Cross compile for Linux, MacOS and Windows on CI Jul 9, 2023

itchyny approved these changes Jul 9, 2023

View reviewed changes

itchyny merged commit ce3701f into jqlang:master Jul 9, 2023

itchyny linked an issue Jul 9, 2023 that may be closed by this pull request

Provide darwin/arm64 builds of jq #2386

Closed

itchyny mentioned this pull request Jul 10, 2023

Build Docker image using pre-built executables #2686

Merged

This was linked to issues Jul 10, 2023

Provide ARM builds #2172

Closed

Provide linux/arm64 binaries in GitHub releases #2591

Closed

Require binary for ARM64 architecture #2275

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cross compile for Linux, MacOS and Windows on CI #2665

Cross compile for Linux, MacOS and Windows on CI #2665

owenthereal commented Jul 7, 2023 •

edited

Loading

wader Jul 7, 2023

nicowilliams Jul 7, 2023

owenthereal Jul 9, 2023 •

edited

Loading

wader Jul 7, 2023

nicowilliams Jul 7, 2023

owenthereal Jul 9, 2023

wader commented Jul 7, 2023

wader commented Jul 7, 2023

wader commented Jul 7, 2023

wader commented Jul 7, 2023

itchyny Jul 7, 2023

owenthereal Jul 9, 2023

userdocs commented Jul 7, 2023

itchyny commented Jul 8, 2023 •

edited

Loading

itchyny Jul 8, 2023

owenthereal commented Jul 9, 2023 •

edited

Loading

userdocs commented Jul 9, 2023

userdocs commented Jul 9, 2023

owenthereal commented Jul 9, 2023

itchyny left a comment

wader commented Jul 9, 2023

Cross compile for Linux, MacOS and Windows on CI #2665

Cross compile for Linux, MacOS and Windows on CI #2665

Conversation

owenthereal commented Jul 7, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

owenthereal Jul 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wader commented Jul 7, 2023

wader commented Jul 7, 2023

wader commented Jul 7, 2023

wader commented Jul 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

userdocs commented Jul 7, 2023

itchyny commented Jul 8, 2023 • edited Loading

Choose a reason for hiding this comment

owenthereal commented Jul 9, 2023 • edited Loading

userdocs commented Jul 9, 2023

userdocs commented Jul 9, 2023

owenthereal commented Jul 9, 2023

itchyny left a comment

Choose a reason for hiding this comment

wader commented Jul 9, 2023

owenthereal commented Jul 7, 2023 •

edited

Loading

owenthereal Jul 9, 2023 •

edited

Loading

itchyny commented Jul 8, 2023 •

edited

Loading

owenthereal commented Jul 9, 2023 •

edited

Loading