Reduce Antrea binary sizes

[antoninbas]

@tnqn pointed out at the Antrea community meeting that the antctl binary which we include in the Antrea Docker image (as well as in release assets) is pretty large, almost 100MB, compared to binaries from other projects (for example, kubectl is only 50MB, yet packs more "features").

K8s uses https://github.com/kubernetes/kubernetes/blob/master/hack/lib/golang.sh to build production binaries, and it is probably a good reference.

A quick experiment revealed that building antctl with go build --ldflags "-s -w", which removes debug symbols, reduced the size of antctl-linux by around 30%.

before:

-rwxr-xr-x@ 1 abas  staff  94406832 Jan  4 07:20 antctl-darwin
-rwxr-xr-x@ 1 abas  staff  82919625 Jan  4 07:21 antctl-linux
-rwxr-xr-x@ 1 abas  staff  84231168 Jan  4 07:22 antctl-windows.exe

after:

-rwxr-xr-x@ 1 abas  staff  74835768 Jan 16 21:25 antctl-darwin
-rwxr-xr-x@ 1 abas  staff  57417728 Jan 16 21:25 antctl-linux
-rwxr-xr-x@ 1 abas  staff  59106304 Jan 16 21:26 antctl-windows.exe

This is a pretty significant gain. The obvious drawback is that if antctl panics "in the field", we won't have a useful traceback to use for troubleshooting (not accurate: we still get useful stack traces). However, it is very common to trim debug symbols from production binaries.

Removing debug symbols is a pretty low-hanging fruit. There may be other things we can do to further reduce binary sizes. Use this issue to document investigations.

• Remove debug symbols from antctl release binaries (we already remove them from release assets, but not from antctl binaries included in container images). We could also remove debug symbols from other Antrea binaries (antrea-agent, antrea-controller), but that may impact our ability to troubleshoot issues in live clusters with no significant drawback (see below)..
• Remove all file system paths by building release binaries with -trimpath. Unlikely to reduce the size much, but this is good practice. There should be no drawback.
• Investigate whether some dependencies / imports are responsible for a significant size increase.

[antoninbas]

I was incorrect before: when using --ldflags "-s -w", we still preserve function names and line numbers. So stack traces for example still provide useful information. A lot of go tool pprof features are still accessible too (e.g. to inspect memory profiles), bunctionality that depends on access to the source code (e.g. the list command) will require a local copy of the source code. Using these ldflags will also restrict the ability to use a debugger, which may not be as much as a concern (delve may also be able to use a local copy of the source code?).

On a separate not, I have noticed the following discrepancy: the antctl binaries included in release assets have debug symbols stripped while the antctl binaries included in Docker images do not.

[tnqn]

On a separate not, I have noticed the following discrepancy: the antctl binaries included in release assets have debug symbols stripped while the antctl binaries included in Docker images do not.

It seems the discrepancy has been there for a long time: #517 stripped the binary in release asset since it was introduced. Maybe we should just add the flags to image build workflow?

Investigate whether some dependencies / imports are responsible for a significant size increase.

I found there is an unnecessary import causing significant size increase: k8s.io/apiserver/pkg/server, will create a PR for this.

[xliuxu]

Can we leverage the busybox way to combine all antrea executables into one binary?
We can implement a shared main package or just use some existing tools such as https://github.com/u-root/gobusybox/.
I did a quick test and confirmed that with https://github.com/u-root/gobusybox/ we can get a 64MB binary containing antctl, antrea-agent, antrea-cni, and antrea-controller.

makebb antrea.io/antrea/cmd/antrea-agent antrea.io/antrea/cmd/antctl antrea.io/antrea/cmd/antrea-controller antrea.io/antrea/cmd/antrea-cni
03:36:25 Disabling CGO for u-root...
03:36:25 Build environment: GOARCH=amd64 GOOS=linux GOPATH=/home/xliu2/go CGO_ENABLED=0 GO111MODULE= GOROOT=/usr/local/go PATH=/usr/local/go/bin
03:37:55 Successfully built "bb" (size 66994176 bytes -- 64 MiB).

~/antrea$ ls -lath bb
-rwxrwxr-x 1 xliu2 xliu2 64M Mar  1 03:43 bb

~/antrea$ ./bb
Failed to run command: command is not present in busybox: bb
Supported commands are:
 - antctl
 - antrea-agent
 - antrea-cni
 - antrea-controller

[tnqn]

Can we leverage the busybox way to combine all antrea executables into one binary?

@xliuxu thank you for the proposal and the data. I can see the benefit to the image sizes, but it would also increase sizes of smaller binaries which needs to be distributed as individual executables, i.e. antrea-cni and antctl.

• For antrea-cni, its size is 8MB, and needs to be copied to host's filesystem. If we have to copy a combined binary, we have to duplicate 64MB space which basically offsets the benefit, so it doesn't seem good to be included.
• For antctl, its size is 45MB (after Optimize package organization to reduce antctl binary size #6037), and we expect users to download it and use it locally, especially after it supports installation and upgrade. A combined binary will increase it by 20MB.
• For antrea-agent and antrea-controller, they are delivered via two different images anyway, so building them into one isn't really helpful to the image sizes, and may even increase the image sizes a bit.
• The only obviously beneficial combination is antctl + antrea-agent, and antctl + antrea-controller, which basically reduces the size of antctl. However, it also means antctl would be used in more than one way: the standalone antctl mode, the combined binary mode which needs a subcommand to use antctl. I'm not sure if we want to trade the inconstentcies with the saved size.

Besides, the fact that the combined binary is not far larger than antrea-agent and antrea-controller may indicate that each component is almost fully importing the other's code and there should be some spaces to improve.

@antoninbas what's your opinion?

[antoninbas]

I feel the same way as @tnqn. That capability is great for programs that are meant to be shipped together most of the time, while preserving the ability to build them as separate binaries if needed. It may be better to put some additional effort into stripping unnecessary dependencies from antctl, antrea-agent and antrea-controller (I would say that antrea-cni is in a good place after the recent changes).

[antoninbas]

We are in pretty good shape for the v2.0 release, and several improvements have been made to reduce binary sizes by re-organizing code to avoid unnecessary imports (#6037, #6038). I think that we should still make one more audit to confirm that we are in a good place and that there is no obvious improvement that can be made, so I will keep the issue open for now, but I will remove it from the v2.0 milestone.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment, or this will be closed in 90 days

[luolanzone]

@antoninbas do you see any TODO for the item Investigate whether some dependencies / imports are responsible for a significant size increase.?

[antoninbas]

It would be nice to check if some imports are responsible for a big chunk of the binary sizes. antctl binaries are still around 50MB, but that's in line with kubectl.
There may be tools to help visualize which dependencies contribute to most of the size.
However, at this stage I don't think there any low-hanging fruits that can help us easily reduce the size by a lot.