Enable envoy images build on Arm CI environments #11813

Jingzhao123 · 2020-06-30T09:16:37Z

In this patch, it will enable the envoyproxy/envoy arm image to build
in community arm CI environments.

Do some modifications in docker_ci.sh script for building arm images by buildx. It will firstly set up environments. Then use the
buildx tool to build the envoyproxy/envoy arm images on x86 platform.
Modify the docker build job for building multi-arch images. It will firstly download the arm64 and amd64 envoy binaries. Then
invoke the docker_ci.sh scripts to generate images.

Signed-off-by: Jingzhao [email protected]

For an explanation of how to fill out the fields, please see the relevant section
in PULL_REQUESTS.md

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
In this PR, we enable the envoyproxy/envoy arm64 image on Envoy CI system.

For envoyproxy/envoy arm64 images, envoy-alpine, envoy-alpine-debug and envoy-google-vrp still do not support on Arm64 platform since some technical reasons. Only the envoyproxy/envoy is enabled on CI system.
In this PR, the buildx tool was used for building multi-arch images on x86 platform.
In addition, the display name of building images job is modified from "Linux-x64 docker" to "Linux multi-arch docker".

[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Deprecated:]

ci/docker_ci.sh

lizan · 2020-06-30T22:44:00Z

.azure-pipelines/pipelines.yml

+    displayName: "Linux-arm64 docker"
+    dependsOn: ["release_arm64"]
+    condition: and(succeeded(), eq(variables['PostSubmit'], 'true'), ne(variables['Build.Reason'], 'PullRequest'))
+    pool: "arm-large"


Can we do this in existing docker job with multiarch?

Hi, lizan
Yes, we can. And the buildx tool could make code more clean. But, the build image by qemu may be a little bit slow, i don't know whether there is enough time to do this. So I choose to build it on the arm environment.

What is your opinion about it?

I'm leaning to do it with buildx, I don't think you'll face any issue, we just do base image + putting binary in it.

Sound good! I will cancel this PR.

Hi, I update this patch by buildx tool.

Hi, Could you please give some comments @lizan

lizan · 2020-07-14T21:45:13Z

ci/Dockerfile-envoy

@@ -31,9 +31,16 @@ RUN adduser --group --system envoy

 RUN mkdir -p /etc/envoy

-ADD build_release_stripped/envoy /usr/local/bin/envoy
+ADD arm64/build_release_stripped/envoy /usr/local/bin/envoy-arm64


Effectively you put everything in the image here, i.e. making the image big, doing rm later doesn't remove the binary form docker layers. Can this actually just be a build-arg?

Has updated. Please check it.

lizan · 2020-07-15T23:36:02Z

ci/Dockerfile-envoy-google-vrp

@@ -1,4 +1,41 @@
-FROM envoyproxy/envoy:local
+ARG BUILD_FROM=ubuntu:18.04


Why do you need this change? you don't really need this?

Hi, Lizan
The envoy-google-vrp is based on envoyproxy/envoy:local image. But when building envoy-google-vrp by buildx, I found that the buildx can not find the envoyproxy/envoy:local image which was just built by buildx. The envoyproxy/envoy:local image does not exist in docker images list.
If we export envoyproxy/envoy:local by command "docker buildx build -o type=docker ", buildx still can not access that image. Although "docker build" could access envoyproxy/envoy:local image.
So, I think, if we replace based image by source dockerfile, the build script may be simpler than before. At same time, for envoy-google-vrp image, it could be more dependent. And it also does not extend the building time since it will use the image cache which was built before.

Will defer this part to @htuch

Seem plausible. FWIW, I don't think we need VRP for ARM, but if you'd like to cleanup the build flow while retaining the same semantics (i.e. have the equivalent of envoyproxy/envoy latest state based on the current tree as the base) then that's great.

@lizan Have updated. Please check it.

In this patch, it will enable the envoyproxy/envoy arm image to build in community arm CI environments. 1. Do some modifications in docker_ci.sh script for building arm images by buildx. It will firstly set up environments. Then use the buildx tool to build the envoyproxy/envoy arm images on x86 platform. 2. Modify the docker build job for building multi-arch images. It will firstly download the arm64 and amd64 envoy binaries. Then invoke the docker_ci.sh scripts to generate images. Signed-off-by: Jingzhao <[email protected]>

1. Update building scripts 2. Update dockerfiles for building arm64 images Signed-off-by: Jingzhao.Ni <[email protected]>

.azure-pipelines/pipelines.yml

Signed-off-by: Jingzhao.Ni <[email protected]>

lizan · 2020-07-29T04:16:47Z

.azure-pipelines/pipelines.yml

      - bash: |
          set -e
-          tar zxf $(Build.StagingDirectory)/bazel.release/envoy_binary.tar.gz
+          mkdir -p linux/amd64 && tar zxf $(Build.StagingDirectory)/bazel.release/envoy_binary.tar.gz -C ./linux/amd64
+          mkdir -p linux/arm64 && tar zxf $(Build.StagingDirectory)/bazel.release.server_only.arm64/envoy_binary.tar.gz -C ./linux/arm64


bazel.release instead of bazel.release.server_only.

Signed-off-by: Jingzhao.Ni <[email protected]>

lizan

Thanks, this LGTM. I triggered a post-submit job based on this:
https://dev.azure.com/cncf/envoy/_build/results?buildId=46427&view=results

If this succeeds we can merge with fixing the nit. If no let's address it.

lizan · 2020-07-29T09:00:05Z

ci/docker_ci.sh

 done

+


nit: remove this extra line.

Signed-off-by: Jingzhao.Ni <[email protected]>

lizan · 2020-07-29T21:11:23Z

@Jingzhao123 one last thing, can you add a release note? Thanks.

Jingzhao123 · 2020-07-30T01:36:58Z

@Jingzhao123 one last thing, can you add a release note? Thanks.

Does the release note describe what does the PR do? Or others?

Jingzhao123 · 2020-07-30T01:48:14Z

@Jingzhao123 one last thing, can you add a release note? Thanks.

Does the release note describe what does the PR do? Or others?

Has updated. Please check it.

lizan · 2020-07-30T01:50:19Z

@Jingzhao123 I meant adding a line to https://github.com/envoyproxy/envoy/blob/master/docs/root/version_history/current.rst

Signed-off-by: Jingzhao.Ni <[email protected]>

mattklein123 · 2020-07-30T17:42:19Z

@Jingzhao123 @lizan would it be possible to document the ARM images that we offer here? https://www.envoyproxy.io/docs/envoy/latest/install/building#pre-built-binaries

It's not completely clear to me whether these show up in different repos, are a single multi-arch image, how a user consumes them, etc. Thank you!

In this patch, it will enable the envoyproxy/envoy arm image to build in community arm CI environments. 1. Do some modifications in docker_ci.sh script for building arm images by buildx. It will firstly set up environments. Then use the buildx tool to build the envoyproxy/envoy arm images on x86 platform. 2. Modify the docker build job for building multi-arch images. It will firstly download the arm64 and amd64 envoy binaries. Then invoke the docker_ci.sh scripts to generate images. Risk Level: Medium (of breaking images) Testing: CI Docs Changes: N/A Release Notes: Added Fixes envoyproxy#1861 Signed-off-by: Jingzhao.Ni <[email protected]>

In this patch, it will enable the envoyproxy/envoy arm image to build in community arm CI environments. 1. Do some modifications in docker_ci.sh script for building arm images by buildx. It will firstly set up environments. Then use the buildx tool to build the envoyproxy/envoy arm images on x86 platform. 2. Modify the docker build job for building multi-arch images. It will firstly download the arm64 and amd64 envoy binaries. Then invoke the docker_ci.sh scripts to generate images. Risk Level: Medium (of breaking images) Testing: CI Docs Changes: N/A Release Notes: Added Fixes envoyproxy#1861 Signed-off-by: Jingzhao.Ni <[email protected]> Signed-off-by: chaoqinli <[email protected]>

yanavlasov assigned lizan Jun 30, 2020

lizan reviewed Jun 30, 2020

View reviewed changes

lizan added the waiting label Jun 30, 2020

Jingzhao123 force-pushed the EnableArmImageBuild branch from 43ed848 to b79bd57 Compare July 9, 2020 03:22

repokitteh-read-only bot removed the waiting label Jul 9, 2020

Jingzhao123 force-pushed the EnableArmImageBuild branch 6 times, most recently from d8442d8 to 00dc595 Compare July 13, 2020 08:57

lizan reviewed Jul 14, 2020

View reviewed changes

Jingzhao123 force-pushed the EnableArmImageBuild branch from 00dc595 to 6790431 Compare July 15, 2020 10:07

lizan suggested changes Jul 15, 2020

View reviewed changes

Jingzhao added 2 commits July 24, 2020 06:10

Update building script and dockerfile for arm64

0d668ef

1. Update building scripts 2. Update dockerfiles for building arm64 images Signed-off-by: Jingzhao.Ni <[email protected]>

Jingzhao123 force-pushed the EnableArmImageBuild branch from 6a5a6db to 0d668ef Compare July 24, 2020 06:31

lizan suggested changes Jul 24, 2020

View reviewed changes

.azure-pipelines/pipelines.yml Outdated Show resolved Hide resolved

Fix artifactName name in docker build job

a3f1a65

Signed-off-by: Jingzhao.Ni <[email protected]>

lizan reviewed Jul 29, 2020

View reviewed changes

Remove "server_only" wrong label in pipeline file

aadb8aa

Signed-off-by: Jingzhao.Ni <[email protected]>

lizan reviewed Jul 29, 2020

View reviewed changes

ci/docker_ci.sh Outdated

done

Copy link

Member

lizan Jul 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: remove this extra line.

Remove the extra line.

87774ec

Signed-off-by: Jingzhao.Ni <[email protected]>

Jingzhao.Ni added 2 commits July 30, 2020 10:22

Add comments in current release notes

10bdf32

Signed-off-by: Jingzhao.Ni <[email protected]>

Modify order of new items

54aa26a

Signed-off-by: Jingzhao.Ni <[email protected]>

lizan approved these changes Jul 30, 2020

View reviewed changes

lizan merged commit 9d70da7 into envoyproxy:master Jul 30, 2020

Jingzhao123 deleted the EnableArmImageBuild branch July 31, 2020 03:24

shaneutt mentioned this pull request Sep 12, 2020

ARM64/Multi-Arch Image emissary-ingress/emissary#2121

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable envoy images build on Arm CI environments #11813

Enable envoy images build on Arm CI environments #11813

Jingzhao123 commented Jun 30, 2020 •

edited

Loading

lizan Jun 30, 2020

Jingzhao123 Jul 1, 2020

Jingzhao123 Jul 1, 2020

lizan Jul 1, 2020

Jingzhao123 Jul 1, 2020

Jingzhao123 Jul 3, 2020

Jingzhao123 Jul 14, 2020

lizan Jul 14, 2020

Jingzhao123 Jul 15, 2020

lizan Jul 15, 2020

Jingzhao123 Jul 16, 2020

lizan Jul 20, 2020

htuch Jul 20, 2020

Jingzhao123 Jul 23, 2020

lizan Jul 29, 2020

lizan left a comment

lizan Jul 29, 2020

lizan commented Jul 29, 2020

Jingzhao123 commented Jul 30, 2020

Jingzhao123 commented Jul 30, 2020

lizan commented Jul 30, 2020

mattklein123 commented Jul 30, 2020

		@@ -1,4 +1,41 @@
		FROM envoyproxy/envoy:local
		ARG BUILD_FROM=ubuntu:18.04

Enable envoy images build on Arm CI environments #11813

Enable envoy images build on Arm CI environments #11813

Conversation

Jingzhao123 commented Jun 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lizan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lizan commented Jul 29, 2020

Jingzhao123 commented Jul 30, 2020

Jingzhao123 commented Jul 30, 2020

lizan commented Jul 30, 2020

mattklein123 commented Jul 30, 2020

Jingzhao123 commented Jun 30, 2020 •

edited

Loading