[arm]Support compile sonic arm image on arm server #7285

guxianghong · 2021-04-10T17:30:26Z

Why I did it

Support compile sonic arm image on arm server. If arm image compiling is executed on arm server instead of using qemu mode on x86 server, compile time can be saved significantly.
Add kernel argument systemd.unified_cgroup_hierarchy=0 for upgrade systemd to version 247, according to [debian] install systemd version 247 from buster-backports #7228

How I did it

When it is detected that the arm compiling is executed on the arm server, the emulated mode of compiling is skipped. If the arm compiling isn't executed on arm server, the emulated mode should work normally.

How to verify it

build arm sonic image on arm server
build arm sonic image on x86 server with emulated mode
build x86 sonic image on x86 server

Which release branch to backport (provide reason below if selected)

201811
201911
202006
202012

Description for the changelog

if [[ $CONFIGURED_ARCH == armhf || $CONFIGURED_ARCH == arm64 ]]; then
    sudo LANG=C DEBIAN_FRONTEND=noninteractive chroot $FILESYSTEM_ROOT apt-get -y install libcairo2-dev libdbus-1-dev libgirepository1.0-dev libsystemd-dev=247.3-3~bpo10+1 pkg-config
else
    sudo LANG=C DEBIAN_FRONTEND=noninteractive chroot $FILESYSTEM_ROOT apt-get -y install libcairo2-dev libdbus-1-dev libgirepository1.0-dev libsystemd-dev pkg-config
fi

Above modify is for the compile error " libsystemd-dev : Depends: libsystemd0 (= 241-7~~deb10u7) but 247.3-3~~bpo10+1 is to be installed". This error is related to PR #7228, and the d.

A picture of a cute animal (not mandatory but encouraged)

lguohan · 2021-04-12T05:57:28Z

i feel we need to name those docker properly

sonic-slave-buster -> amd64
sonic-slave-buster-arm64 -> emulated arm64 docker image that runs on amd64

for native arm64 image, I feel we should name the image as sonic-slave-buster instead of sonic-slave-buster-amd64. to align with
docker multi-arch document, https://docs.docker.com/docker-for-mac/multi-arch/, we should use manifest to differentiate sonic-slave-buster that runs on arm64, amd64, armhf. https://docs.docker.com/engine/reference/commandline/manifest/

In the docker register, sonic-slave-buster will have three images, one for each platform, amd64/armhf/arm64. when user pull the image, it will pull the correct one based on the client machine arch.

lguohan · 2021-04-12T06:05:03Z

as you know, we introduce this ENABLE_DOCKER_BASE_PULL to pull sonic-slave docker from ACR (azure container registery), and it looks like acr supports docker manifest. https://docs.microsoft.com/en-us/azure/container-registry/container-registry-image-formats which allow registries to store multi-architecture images under a single image:tag reference.

therefore, we should think our naming properly so that when user try to build arm64 natively, they can pull the proper sonic-slave container from acr.

here is docker manifest example:

$ docker manifest create 45.55.81.106:5000/coolapp:v1 \
    45.55.81.106:5000/coolapp-ppc64le-linux:v1 \
    45.55.81.106:5000/coolapp-arm-linux:v1 \
    45.55.81.106:5000/coolapp-amd64-linux:v1 \
    45.55.81.106:5000/coolapp-amd64-windows:v1

Created manifest list 45.55.81.106:5000/coolapp:v1

to fulfill our goal above, we can replace coolapp to sonic-slave-buster.

then, we have following native images
sonic-slave-buster-amd64 for amd64
sonic-slave-buster-arm64 for arm64
sonic-slave-buster-armhf for armhf image.

we can rename current multiarch image to sonic-slave-buster-march-amd64-arm64. it means this is a multiarch image that emulate arm64 on amd64. since we might have a emulated image for armhf on arm64?

if we do not want to rename current multiarch image, then we will rename the native images to
sonic-slave-buster-{amd64,arm64,armhf}-native

should we rename current multiarch image, or should we add native to new native arm64 image?

files/build_templates/sonic_debian_extension.j2

lguohan · 2021-04-12T06:22:45Z

Makefile.work

@@ -98,8 +107,14 @@ endif

 ifeq ($(CONFIGURED_ARCH),amd64)
 SLAVE_BASE_IMAGE = $(SLAVE_DIR)
+EMULATED_COMPILE = n


i am not sure how well known this emulated_compile term is. can we rename to MULTIARCH_QEMU_ENVIRON = y? It is perhaps more precise and better known.

Obviously, MULTIARCH_QEMU_ENVIRON is more precise and better known, I will rename it.

files/build_templates/sonic_debian_extension.j2

Makefile.work

build_debian.sh

lguohan · 2021-04-12T06:26:09Z

build_debian.sh

@@ -581,7 +581,7 @@ sudo mksquashfs $FILESYSTEM_ROOT $FILESYSTEM_SQUASHFS -e boot -e var/lib/docker

 scripts/collect_host_image_version_files.sh $TARGET_PATH $FILESYSTEM_ROOT

-if [[ $CONFIGURED_ARCH == armhf || $CONFIGURED_ARCH == arm64 ]]; then
+if [ $EMULATED_COMPILE == y ] && [[ $CONFIGURED_ARCH == armhf || $CONFIGURED_ARCH == arm64 ]]; then


is 2nd and 3rd condition needed?

files/build_templates/sonic_debian_extension.j2

lguohan · 2021-04-12T06:27:21Z

scripts/build_debian_base_system.sh

-        # qemu arm bin executable for cross-building
-        sudo mkdir -p $FILESYSTEM_ROOT/usr/bin
-        sudo cp /usr/bin/qemu*static $FILESYSTEM_ROOT/usr/bin || true
+        if [ $EMULATED_COMPILE == y ]; then


why line 22 is still needed?

Line 28 is only for arm, then line 22 should be kept.

guxianghong · 2021-04-12T09:53:51Z

as you know, we introduce this ENABLE_DOCKER_BASE_PULL to pull sonic-slave docker from ACR (azure container registery), and it looks like acr supports docker manifest. https://docs.microsoft.com/en-us/azure/container-registry/container-registry-image-formats which allow registries to store multi-architecture images under a single image:tag reference.

therefore, we should think our naming properly so that when user try to build arm64 natively, they can pull the proper sonic-slave container from acr.

here is docker manifest example:
$ docker manifest create 45.55.81.106:5000/coolapp:v1 \
    45.55.81.106:5000/coolapp-ppc64le-linux:v1 \
    45.55.81.106:5000/coolapp-arm-linux:v1 \
    45.55.81.106:5000/coolapp-amd64-linux:v1 \
    45.55.81.106:5000/coolapp-amd64-windows:v1

Created manifest list 45.55.81.106:5000/coolapp:v1
to fulfill our goal above, we can replace coolapp to sonic-slave-buster.

then, we have following native images
sonic-slave-buster-amd64 for amd64
sonic-slave-buster-arm64 for arm64
sonic-slave-buster-armhf for armhf image.

we can rename current multiarch image to sonic-slave-buster-march-amd64-arm64. it means this is a multiarch image that emulate arm64 on amd64. since we might have a emulated image for armhf on arm64?

if we do not want to rename current multiarch image, then we will rename the native images to
sonic-slave-buster-{amd64,arm64,armhf}-native

should we rename current multiarch image, or should we add native to new native arm64 image?

Based on you comments and my research, I suggest to name the slave docker as follows:

Name the native image as sonic-slave-buster, which is same as current definition
Rename march image as sonic-slave-buster-march-{amd64,arm64,armhf}, or keep current name which is sonic-slave-buster-{amd64,arm64,armhf}. For the host arch is clearly when we do compile, there is not need to add host arch to the march image name. I think that add "march" to the name is more precise for understand. I'm not sure whether emulated compile can work on ARM server. When I start march docker by the CLI "docker run --rm --privileged multiarch/qemu-user-static --reset -p yes"(https://github.com/multiarch/qemu-user-static) on arm server, an error return with "standard_init_linux.go:211: exec user process caused "exec format error". I don't know if the AWS vServer doesn't support it or the ARM server doesn't support it.

lguohan · 2021-04-13T23:17:36Z

if you look at the example, the manifest name needs to be different from the image name, so if the manifest is sonic-slave-buster, then image name needs to be different.

lguohan · 2021-04-13T23:20:21Z

scripts/build_debian_base_system.sh

-        # qemu arm bin executable for cross-building
-        sudo mkdir -p $FILESYSTEM_ROOT/usr/bin
-        sudo cp /usr/bin/qemu*static $FILESYSTEM_ROOT/usr/bin || true
+        if [ $MULTIARCH_QEMU_ENVIRON == y ]; then


can you remove line 22?

Reference to https://www.debian.org/mirror/list, debian-archive.trafficmanager.net doesn't have the arm source mirror. Is the http://deb.debian.org/debian used for the source of x86 acceptable?

…rade systemd to version 247

azure-pipelines · 2021-04-15T08:28:54Z

Commenter does not have sufficient privileges for PR 7285 in repo Azure/sonic-buildimage

xumia · 2021-04-15T09:07:47Z

@guxianghong , could you please try "/azpw run" to rerun checkers?
Or rerun the special checker '/azpw run "Azure.sonic-buildimage (Build mellanox)"'
The command /azpw can only used by PR author.

guxianghong · 2021-04-15T09:39:47Z

/azpw run "Azure.sonic-buildimage (Build mellanox)"

mssonicbld · 2021-04-15T09:39:48Z

/AzurePipelines run "Azure.sonic-buildimage (Build mellanox)"

azure-pipelines · 2021-04-15T09:39:53Z

No pipelines are associated with this pull request.

guxianghong · 2021-04-17T05:15:46Z

/azpw run kvmtest-t0

mssonicbld · 2021-04-17T05:15:47Z

/AzurePipelines run kvmtest-t0

azure-pipelines · 2021-04-17T05:15:52Z

No pipelines are associated with this pull request.

guxianghong · 2021-04-17T05:16:54Z

/azpw run kvmtest-t0

mssonicbld · 2021-04-17T05:16:55Z

/AzurePipelines run kvmtest-t0

azure-pipelines · 2021-04-17T05:17:00Z

No pipelines are associated with this pull request.

guxianghong · 2021-04-17T05:19:09Z

/azpw run "Azure.sonic-buildimage (Test vstest)"

mssonicbld · 2021-04-17T05:19:10Z

/AzurePipelines run "Azure.sonic-buildimage (Test vstest)"

azure-pipelines · 2021-04-17T05:19:15Z

No pipelines are associated with this pull request.

guxianghong · 2021-04-17T05:31:22Z

/azpw run "Azure.sonic-buildimage (Test kvmtest-t0)"

mssonicbld · 2021-04-17T05:31:23Z

/AzurePipelines run "Azure.sonic-buildimage (Test kvmtest-t0)"

azure-pipelines · 2021-04-17T05:31:28Z

No pipelines are associated with this pull request.

guxianghong · 2021-04-17T06:06:08Z

/azpw run

mssonicbld · 2021-04-17T06:06:09Z

/AzurePipelines run

azure-pipelines · 2021-04-17T06:06:24Z

Azure Pipelines successfully started running 1 pipeline(s).

dflynn-Nokia · 2021-04-21T15:09:55Z

Any progress on backporting this to 202012?
It seems we will also need to backport PR #5935 to satisfy dependencies.

lguohan · 2021-04-21T15:35:21Z

we are planning to do it, but there were some issue identified for some dependency pr. the pr is not clean as it did the systemd upgrade to 247, however, there is some issue #7364 associated with systemd 247 upgrade, and thus it is blocked.

- Support compile sonic arm image on arm server. If arm image compiling is executed on arm server instead of using qemu mode on x86 server, compile time can be saved significantly. - Add kernel argument systemd.unified_cgroup_hierarchy=0 for upgrade systemd to version 247, according to #7228 - rename multiarch docker to sonic-slave-${distro}-march-${arch} Co-authored-by: Xianghong Gu <[email protected]> Co-authored-by: Shi Lei <[email protected]>

- Support compile sonic arm image on arm server. If arm image compiling is executed on arm server instead of using qemu mode on x86 server, compile time can be saved significantly. - Add kernel argument systemd.unified_cgroup_hierarchy=0 for upgrade systemd to version 247, according to sonic-net#7228 - rename multiarch docker to sonic-slave-${distro}-march-${arch} Co-authored-by: Xianghong Gu <[email protected]> Co-authored-by: Shi Lei <[email protected]>

guxianghong requested review from lguohan, qiluo-msft and xumia as code owners April 10, 2021 17:30

guxianghong force-pushed the xgu-develop-for-arm-compile-20210409 branch from 9dc0759 to 2690e5f Compare April 12, 2021 03:56