Provide an official ARM docker image and/or provide documentation for building on ARM

[gizmochief7]

Would it be possible to get an official Docker image of Envoy on ARM or at least some documentation on how to build it?
My team and I have tried off and on to build Envoy on ARM using the Bazel build process without any luck. Its challenging to say the least. I just saw a PR #1795 by @costinm dated a couple weeks ago based on the issue #1781 that talked about building it on ARM successfully after a few fixes. This is great news but we aren't sure how to duplicate it. @costinm mentioned that he used a custom CMAKE and a pi-cross-compiler to make it work. Documentation on this would be very helpful. Also, if there exist an image I could use for testing, then that would be very helpful too. Thanks again for such an awesome product!

[vielmetti]

@gizmochief7 it's been a little while, have you made progress? It sounds like you're building for 32-bit ARMv7; my particular interest is in arm64 builds.

[gizmochief7]

@vielmetti It has been and I almost forgot about posting this. We did not have any further luck with this attempt and without any community help we decided to move past Envoy. We tried out another reverse proxy/side car project for micro services and it just worked out of the box on both 32-bit and 64-bit. So we haven't wasted any more time here. Sorry I couldn't be more help.

[costinm]

Sorry, didn't see this bug: my cmake files are here: https://github.com/costinm/istio-build (tested on pi and android), I'm using them with repo to get the deps ( start from https://github.com/costinm/istio-repo ) There is another PR needed to compile head ( another option with 64bit int ) - will send it when I get back from vacation. I've been running envoy on my home pi (for a personal server) for few weeks, it's pretty stable and light, but I haven't made any progress in automating the build, CircleCI doesn't seem to like my build.

…

On Mon, Nov 20, 2017 at 8:44 PM, gizmochief7 ***@***.***> wrote: @vielmetti <https://github.com/vielmetti> It has been and I almost forgot about posting this. We did not have any further luck with this attempt and without any community help we decided to move past Envoy. We tried out another reverse proxy/side car project for micro services and it just worked out of the box on both 32-bit and 64-bit. So we haven't wasted any more time here. Sorry I couldn't be more help. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1861 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFI6jvr_r5AMUuCa4cAUnkmziR4N3x5ks5s4lUkgaJpZM4P5-S8> .

[rshriram]

fwiw, I tried to build bazel on my pi (yes, it took several hours), and then built envoy on pi for armv7l 64bit. While envoy started up and was able to talk to http services, it had difficulty talking to certain https services (esp twitter). There seemed to be some boringssl issue that prevented ssl connections from working. My hunch was that it didn't compile properly. At that point, I gave up, because I had spent 4 solid days fighting bazel on pi.

[costinm]

I think the cross-compilation option is faster and less painful - and easy to expand to other platforms. Boringssl and most of the dependencies have native support for cmake (or can be easily adapted) - one of the problems with building with 'bazel' on pi is that deps are not really built with bazel, so even if bazel can cross-compile c++ code, it would still not help us, most of the code comes from the libraries compiled with autoconf. Of course autoconf also supports cross - but then I also wanted to run it on android, and cmake is supported very nicely by the NDK. It requires some periodic tweaking - adding the new envoy files and fixing ocasional 32-bit regressions (so far only the flags package seems to be a problem - not clear why), but it's a pretty fast build. But I appreciate the pain and effort in straight-compiling and fighting bazel :-)

…

On Tue, Nov 21, 2017 at 6:35 AM, Shriram Rajagopalan < ***@***.***> wrote: fwiw, I tried to build bazel on my pi (yes, it took several hours), and then built envoy on pi for armv7l 64bit. While envoy started up and was able to talk to http services, it had difficulty talking to certain https services (esp twitter). There seemed to be some boringssl issue that prevented ssl connections from working. My hunch was that it didn't compile properly. At that point, I gave up, because I had spent 4 solid days fighting bazel on pi. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1861 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFI6mUOLKwygzX6Xwx9LXYCxrEc7_oEks5s4t-fgaJpZM4P5-S8> .

[costinm]

https://github.com/costinm/istio-build/releases if you want to try some (older) binaries, matching ~ Istio 0.2

…

On Tue, Nov 21, 2017 at 9:23 AM, Costin Manolache ***@***.***> wrote: I think the cross-compilation option is faster and less painful - and easy to expand to other platforms. Boringssl and most of the dependencies have native support for cmake (or can be easily adapted) - one of the problems with building with 'bazel' on pi is that deps are not really built with bazel, so even if bazel can cross-compile c++ code, it would still not help us, most of the code comes from the libraries compiled with autoconf. Of course autoconf also supports cross - but then I also wanted to run it on android, and cmake is supported very nicely by the NDK. It requires some periodic tweaking - adding the new envoy files and fixing ocasional 32-bit regressions (so far only the flags package seems to be a problem - not clear why), but it's a pretty fast build. But I appreciate the pain and effort in straight-compiling and fighting bazel :-) On Tue, Nov 21, 2017 at 6:35 AM, Shriram Rajagopalan < ***@***.***> wrote: > fwiw, I tried to build bazel on my pi (yes, it took several hours), and > then built envoy on pi for armv7l 64bit. While envoy started up and was > able to talk to http services, it had difficulty talking to certain https > services (esp twitter). There seemed to be some boringssl issue that > prevented ssl connections from working. My hunch was that it didn't compile > properly. At that point, I gave up, because I had spent 4 solid days > fighting bazel on pi. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#1861 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AAFI6mUOLKwygzX6Xwx9LXYCxrEc7_oEks5s4t-fgaJpZM4P5-S8> > . >

[moderation]

I've secured a Pine64 Rock64 SOC which is ARM8 / aarch64 and has 4GB of RAM. I've successfully
compiled Bazel 0.13.0. I'm not having any luck compiling Envoy.

I've leveraged @clnperez way of removing the Lua JIT.

I can't get her trick to use the locally installed Go to work go_register_toolchains(go_version="host"). When I use this Bazel uses a symbolic link to the installed Go tools (~/Library/go/pkg/tool/linux_arm64) and tries to compile Gazelle and protoc-gen-go and fails because link can't see gcc on $PATH. Installing these tools on the host don't fix the issue. I expect this is something to do with breaking the hermetic seal by using go_version="host".

I've modified source/extensions/extensions_build_config.bzl to only include a minimum number of extensions.

I'm using bazel --bazelrc=/dev/null build -s -c opt //source/exe:envoy-static.stripped --define google_grpc=disabled --define signal_trace=disabled --define hot_restart=disabled --verbose_failures as the build command with the following WORKSPACE.

workspace(name = "envoy")

load("//bazel:repositories.bzl", "envoy_dependencies")
load("//bazel:cc_configure.bzl", "cc_configure")

envoy_dependencies(
   skip_targets=['luajit']
)
cc_configure()

load("@envoy_api//bazel:repositories.bzl", "api_dependencies")
api_dependencies()

load("@io_bazel_rules_go//go:def.bzl", "go_download_sdk", "go_rules_dependencies", "go_register_toolchains")
load("@com_lyft_protoc_gen_validate//bazel:go_proto_library.bzl", "go_proto_repositories")

go_proto_repositories(shared=0)
go_rules_dependencies()

go_download_sdk(name = "go_sdk", sdks = {"linux_arm64":  ("go1.10.2.linux-arm64.tar.gz", "d6af66c71b12d63c754d5bf49c3007dc1c9821eb1a945118bfd5a539a327c4c8"), }, )

go_register_toolchains()

This prevents the build automatically downloading Go for linux_amd64.

The build then fails with

ERROR: ~/.cache/bazel/_bazel_moderation/1d277edd85b03195d275adc4870a9bef/external/com_lyft_protoc_gen_validate/validate/BUILD:35:1: no such package '@com_github_golang_protobuf//ptypes/duration': no such package '@io_bazel_rules_go_repository_tools//': no such package '@go_sdk//': Unsupported host linux_amd64 and referenced by '@com_lyft_protoc_gen_validate//validate:go_default_library'
ERROR: Analysis of target '//source/exe:envoy-static.stripped' failed; build aborted: no such package '@com_github_golang_protobuf//ptypes/duration': no such package '@io_bazel_rules_go_repository_tools//': no such package '@go_sdk//': Unsupported host linux_amd64

My suspicion is that the lyft/protoc-gen-validate is hard coded for linux_amd64 (or considering compilation works on MacOS / Darwin maybe it just doesn't support ARM).

Any help from Bazel wizards like @jmillikin-stripe or from people that have tried non amd64 compilation like @clnperez @costinm @rshriram would be great.

[vielmetti]

Issue noted in bufbuild/protoc-gen-validate#75 - there are definitely some hard-coded dependencies in lyft/protoc-gen-validate to amd64 binaries.

[clnperez]

I'll try and help. I don't have much ARM experience but a coworker (@tophj-ibm) does. So we'll try to put our heads together on this one. Any progress we can make would be great. We'd also like to have a ppc64le docker image.

wonder twins unite?

My first guess is that something in cc_configure works differently for ARM. I ran into some things where, when using certain flags, gcc would behave differently for power than it did for x86. envoy, for the last release, seems to have come to rely a lot more on how bazel configured the gcc toolchain and did a little less customization. That got rid of almost all the issues I was seeing, but maybe there is some more tweaking needed for ARM.

For your WORKSPACE file, what's the logic behind including the call to `go_download_sdk'?

[moderation]

Wonder twins unite @clnperez! Without adding the specific go_download_sdk when trying to build on ARM you can see Bazel download the the amd64 version of Go and then subsequently fails as the Go it has downloaded doesn't work on aarch64. I agree that the gcc wrapping could be problematic but the errors I'm currently seeing are linux_amd64 errors.

[vielmetti]

The issue of Bazel mis-identifying the aarch64 system is noted at bazel-contrib/rules_go#1506 - thanks for the diagnosis.

[moderation]

Before resuming work on this I think we should prioritize moving to rules_go 0.12.0. At the moment a build with 0.12.0 fails with a cycle in dependency graph error. I suspect the root cause for this and the inability to build on aarch64 is lyft/protoc-gen-validate. It has a number of hacks and workarounds in that it works today but I think without some maintenance it will prevent a move to rules_go 0.12.0 and the ability to compile for non amd64, darwin, s390x platforms.

[clnperez]

So, I'm hitting this now as well on ppc64le. :D

[moderation]

@clnperez I've had success this week building on aarch64 / arm64 using these patches from @lubinsz #3681 (comment). @htuch unblocked protobuf 3.6.0 with this patch to lyft/protoc-gen-validate bufbuild/protoc-gen-validate@345b6b4. rules_go 0.12.0 is still blocked.

[clnperez]

@moderation i just took a look at those patches, and they look like they're all adding arm support, whereas we already have power support. i'm actually hitting a different error today than I did last night, and that is the gazelle compile (at least i think it's the gazelle compile) isn't finding gcc in my PATH. I might open a new issue for that.

[moderation]

@clnperez We were having the same issue on arm64. The patch for rules_go at lubinsz/rules_go@497b488 fixed that issue. Not sure how portable that patch is but might give you some ideas.

[clnperez]

@moderation using it as-is seems to break compatibility with lyft/protoc-gen-validate as the patch for rules_go is based off of a branch that deprecated go_repositories. so i tried just changing the path bits for the linking (using what was in that patch) and it didn't get me around that.

there are so many moving parts to trying out one thing, though, i could have missed something. but i think i have it all lined up correctly. i'll dig some more and see what i find.

edit: link the link-only patch based off of rules_go 0.11.1: clnperez/rules_go@2f55b46

[clnperez]

@moderation it turns out this was what i needed: bazel-contrib/bazel-gazelle#242

[moderation]

@clnperez this is similar to this patch that enabled the arm64 build - lubinsz/rules_go@497b488#diff-b88d1275b3ee8565f7821ba59b402139. Thanks @lubinsz. It would be good to go back and have these changes included upstream.

[vielmetti]

Several upstream issues in bazel have been resolved, specifically bazel-contrib/rules_go#1550 which might make this worth a retry.

[jared2501]

Would be super interested in an arm64 docker image!

[trilom]

Put this together but it does not build envoy, something about external/go_sdk/bin location not being found. I've been able to nurse it past but have ran into a couple other issues that I believe might be around environment. Either way this feels pretty close to an alpine build on edge if it's useful to anyone.

FROM alpine:edge AS bazel

ARG BAZEL_VERSION='3.3.1'

RUN mkdir /bazel
WORKDIR /bazel
RUN wget https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/bazel-${BAZEL_VERSION}-dist.zip
RUN unzip bazel-${BAZEL_VERSION}-dist.zip
# script needs bash, build needs rest
RUN apk add bash=5.0.17-r0 \
            openjdk8=8.242.08-r2 \
            build-base=0.5-r2 \
            linux-headers=5.4.5-r1 \
            zip=3.0-r8 \
            python3=3.8.3-r0 
RUN ln -s /usr/bin/python3 /usr/local/bin/python && \
    env JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" bash ./compile.sh

FROM alpine:edge AS envoy
RUN apk add git=2.27.0-r0 \
            openjdk8=8.242.08-r2 \
            autoconf=2.69-r2 \
            automake=1.16.2-r0 \
            libtool=2.4.6-r7 \
            samurai=1.1-r0 \
            cmake=3.17.3-r0 \
            python3=3.8.3-r0 \
            build-base=0.5-r2 \
            llvm10-dev=10.0.0-r2 \
            go=1.14.4-r1 \
            clang=10.0.0-r3 \
            linux-headers=5.4.5-r1 \
            libc6-compat=1.1.24-r9 \
            musl-dev=1.1.24-r9
RUN go get -u github.com/bazelbuild/buildtools/buildifier
RUN go get -u github.com/bazelbuild/buildtools/buildozer
RUN git clone https://github.com/envoyproxy/envoy.git /envoy
WORKDIR /envoy
COPY --from=bazel /bazel/output/bazel /usr/local/bin/bazel

RUN ln -sf /usr/lib/llvm10/ /opt/llvm && \
    ln -sf /usr/bin/python3 /usr/local/bin/python && \
    env JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk \
    bazel build //source/exe:envoy-static --sandbox_debug

[trilom]

ci/run_envoy_docker.sh 'ci/do_ci.sh bazel.release.server_only' no longer works Tick-Tocker/bazel-arm64 does not have Bazel 3.3.1 version and the .bazelversion specifies 3.3.1. If you switch it to 3.3.0 it will build but I am not sure if it is successful yet.

[lizan]

@Jingzhao123 ^^

[lizan]

@trilom It is not recommended to build a musl (alpine) based binary, you have to disable signal_actions and gperftools. Disabling latter will impact performance significantly (around 30~40% on x64 on a simple benchmark)

[trilom]

@trilom It is not recommended to build a musl (alpine) based binary, you have to disable signal_actions and gperftools. Disabling latter will impact performance significantly (around 30~40% on x64 on a simple benchmark)

I did read that at one point, I'm not versed in building with musl but I did read similar, and that is for the most part why I stopped with the alpine approach. Thank you for the information. My ideal goal is to use something similar to docker buildx for this so that I can have some CI to this image (something similar to this), but aside from that the command above should fit my needs.

[Jingzhao123]

@lizan @trilom It has been updated.

[CelsoSantos]

Realistically speaking, how far are we from an arm64 image that could be considered stable?
From my understanding, we already have an image that runs on arm64, that can be used to build Envoy from arm64 architectures. But then does this produce an arm64 image or just the binary that still needs to be containerized?

Also (though, this one is my fault for not knowing), how long does it take to build an arm64 image with that container? Hours, minutes? Days? I have a RPi4 available that I could use to build Envoy.

Also, I'm getting an error trying to pull the images:

pi@k3s:~ $ docker pull envoyproxy/envoy-build-ubuntu:f21773ab398a879f976936f72c78c9dd3718ca1e
f21773ab398a879f976936f72c78c9dd3718ca1e: Pulling from envoyproxy/envoy-build-ubuntu
no matching manifest for linux/arm/v7 in the manifest list entries

EDIT: Nevermind the error.. I forgot I had a 32bit OS on this Pi

[carmiac]

@CelsoSantos I ran the build just a couple of days ago on my RPi4. It produced a static non-containerized binary that I was able to drop onto my target device and run. The build time was around 12 hours, though I did notice that it was just using a single core.

[moderation]

@carmiac the single core issue you saw began with a recent release of Bazel. Here is what I posted in the Bazel Slack.

This line in the Envoy Proxy :bazel: config is the culprit - https://github.com/envoyproxy/envoy/blob/master/.bazelrc#L14. According to the documentation this flag has remained as false as the default. But my compilation went to a single core when I jumped to Bazel 3.2.0. The config in the Envoy code doesn't specify whether it is true or false. My best guess is that in 3.2.0 the interpretation of the flag without a true or false changed and the Envoy Proxy definition flipped to being evaluated as true. The following override in ~/.bazelrc has me back to using multiple concurrent cores.

# disable as this was restricting Bazel actions to 1
build --experimental_local_memory_estimate=false

Setting this flag will allow you to use more than one core. I had a 4G RAM RPi4 and I had to restrict the build using --jobs=3 or --jobs=2 to prevent memory exhaustion. I just upgraded to am 8G RAM RPi4 and no longer have this issue and can use all 4 cores. All of the above assumes arm64 using a 64 bit OS like Ubuntu Server.

Lastly check out https://stevesloka.com/compile-envoy-on-raspberry-pi4/.

[al45tair]

@CelsoSantos It's been a while, but IIRC the command I ran back on the 18th of May worked and generated a Docker image (which I uploaded to Docker Hub as al45tair/envoy-arm64). It seems to work.

@moderation @carmiac It's probably worth getting some time on an ARM64 server machine rather than building on a Pi. Even if you don't have access to one at work (I'm lucky in that regard), you can spin them up in AWS — there are Graviton and Graviton2 instances available, I believe — and they'll build it a lot faster than a Pi will.

[moderation]

@al45tair I'm pretty happy with my local RPi4 dev environment. After the initial build the subsequent incremental builds are relatively quick and compiling out all the extensions you don't need works well. I'm in a very small minority of people who don't think Docker helps with anything so I'm building Bazel and Envoy straight on the RPi4

[lizan]

Realistically speaking, how far are we from an arm64 image that could be considered stable?

I wouldn't be comfortable saying it is stable until it passes most of our test suite on arm64.

[vielmetti]

A note that Bazel 3.4.0 now ships with official arm64 binaries -

bazelbuild/bazel#8833 (comment)

That may save you a build step, @moderation , if you can adapt to any changes associated with the Bazel point release at 3.4.0.

[moderation]

Thanks @vielmetti. Looks like they need to work on their automation a bit as the latest release 3.4.1 isn't yet available. But definitely looks promising