Create release Docker images for stacks-signer binary

[hstove]

Opening this based off this discussion: #4268 (comment)

PR #4268 adds the stacks-signer and other binaries to the Docker images that we create in CI runs. Previously, only the stacks-node binary was included. Including the other binaries helps us publish images with the signer, but it significantly increases the image size (880 MB to 1.6 GB). For regular CI runs, this is ok, because it lets us debug builds via Docker. For releases, however, we may want to either optimize the images or build separate images for the different binaries.

[wileyj]

this may be a little tricky, and i have not tried something like this with rust projects so i'm not sure how much we can shrink the binaries.
there's a few standard methods we can try to shrink binaries, but there may also be some changes that would have to be made in how we compile, i.e. the cargo.toml file.

[wileyj]

related: https://kobzol.github.io/rust/cargo/2024/01/23/making-rust-binaries-smaller-by-default.html

[BowTiedDevOps]

I built the binaries locally and stripped them afterwards, resulting in a significant size decrease.

Command ran: strip --strip-debug <bin>

OS: Debian 13

Size reduction:

• stacks-node: from 358MB to 39MB
• stacks-signer: from 208MB to 11MB

I've also tested the binaries after the strip to ensure that they don't panic and still work, but further testing must be done in order to be sure that they work as expected.

[wileyj]

outstanding, thanks! we'll want to run the binaries though a bit more testing, but this looks very promising and simple change to make

[BowTiedDevOps]

Total size of all packages before reductions: 1545 MB

It looks like stripping the debug-info will be default in a future (1.77) stable release of cargo (currently implemented in nightly): Discussion comment

I found another way to strip, by also stripping all symbols (running just strip <bin>) instead debug-info only. This way of stripping removes too much from the binary, as an example running this command on stacks-signer removes the binary completely. These are the results:

	unstripped	strip debug-info	strip all symbols
stacks-node	374 MB	40 MB	34 MB
stacks-signer	363 MB	12 MB	Removes bin file

By adding strip = "debuginfo" to [profile.release] inside the root Cargo.toml, I’ve gotten the total size of all the binaries to 94.691 MB, however, this strips the debug-info symbols from all the binaries, which may not be desired. In order to only strip specific binaries, you can run strip --strip-debug <bin>...<bin> after the release build, or add package.<package_name>.strip = "debuginfo" to [profile.release].

Size of binaries after the strip:

Using opt-level = "z" for the release profile (along with stripping) gives roughly the same result as simply stripping (stacks-node from 40 MB to 37 MB, stacks-signer - same size). The build time is decreased from 3m 33s to 2m 05s.

opt-level = "s" reduces the binaries’ size by about 1 MB more than opt-level = "z" (stacks-node from 40 MB → 36 MB, stacks-signer from 12 MB to 11 MB). The build time was increased from 3m 33s to 4m 06s.

Enabling link time optimization (lto = true in release profile) makes stackslib compile fail. However, building only specific binaries, stacks-node (6m 03s build) got to 36 MB and stacks-signer (2m 22s build) to 9.3 MB. Combining lto with opt-level = "s" decreases stacks-node to 30 MB and stacks-signer to 7.5 MB.

Reducing the parallel code generation units to 1 (codegen-units = 1) decreases the size of stacks-node to 28 MB and stacks-signer to 7.1 MB.

Aborting instead of unwinding on panic (panic = "abort") decreases stacks-node to 27 MB and stacks-signer to 6.5 MB.

Additions to the root Cargo.toml:

[profile.release]
debug = true
# package.stacks-node.strip = "debuginfo"
# package.stacks-signer.strip = "debuginfo"
strip = "debuginfo"
lto = true
opt-level = "s"
codegen-units = 1
panic = "abort"

It is also possible to parse these settings through the RUSTFLAGS environment variable, so you don’t have to add them inside the Cargo.toml file:

RUSTFLAGS="-C strip=debuginfo -C opt-level=s -C codegen-units=1 -C panic=abort" cargo build --features monitoring_prom,slog_json --release --workspace

Final size (with lto):

• stacks-node - 27 MB
• stacks-signer - 6.5 MB

However, having lto = "true" fails the build on stackslib, so, building without it gives us a final size of:

• stacks-node - 30 MB
• stacks-signer - 8.3 MB
• All binaries - 67.178 MB

Using UPX tool in order to reduce the size of the binaries (upx --best --lzma <bin>...<bin>):

• stacks-node - 15 MB
• stacks-signer - 2.9 MB
• All binaries - 27.612 MB

Size of binaries after all steps (without lto)

To sum it up, I've managed to reduce the total size of all the binaries from 1.545 GB to 67.178 MB (without UPX), or 27.612 MB with the binary compressor. I tested that the binaries still run, that the RUST_LOG_DEBUG=1 env still prints the debug messages, I've started a mock miner node successfully, tested that the stacks-signer get-chunk command works and blockstack-cli decode-tx successfully decodes a raw transaction. Every setting can be passed as env in CLI, so no files have to be modified, and the only thing that seems to be missing is the panic backtrace (which I removed with panic=abort, and it can be included back).

[hstove]

Is the main downside of stripping debug symbols that you can't get backtraces with RUST_BACKTRACE=1 ...?

[wileyj]

yeah there are some extra considerations to consider here vs just stripping the binaries. we'll work on a fork before we pr anything here- i think any change will have a larger effect so we'll want tobe explicit as to why we chose a specific method.

…

On Mon, Jan 29, 2024 at 14:01 Hank Stoever ***@***.***> wrote: Is the main downside of stripping debug symbols that you can't get backtraces with RUST_BACKTRACE=1 ...? — Reply to this email directly, view it on GitHub <#4278 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVXIHBHHYEBAIR6GOMCFUTYRAL3DAVCNFSM6AAAAABCHFVVEWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJVGY2TANZSGI> . You are receiving this because you were assigned.Message ID: ***@***.***>

[wileyj]

one more wrinkle that i've enountered here - it's only possible to add a release binary today for the source arch.
because of how stacks-signer currently builds, every other build fails (i.e. anything cross-compiled).
looking into options here, but it's definitely a blocker

[blockstack-devops]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.