Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create release Docker images for stacks-signer binary #4278

Closed
hstove opened this issue Jan 23, 2024 · 9 comments
Closed

Create release Docker images for stacks-signer binary #4278

hstove opened this issue Jan 23, 2024 · 9 comments
Assignees

Comments

@hstove
Copy link
Contributor

hstove commented Jan 23, 2024

Opening this based off this discussion: #4268 (comment)

PR #4268 adds the stacks-signer and other binaries to the Docker images that we create in CI runs. Previously, only the stacks-node binary was included. Including the other binaries helps us publish images with the signer, but it significantly increases the image size (880 MB to 1.6 GB). For regular CI runs, this is ok, because it lets us debug builds via Docker. For releases, however, we may want to either optimize the images or build separate images for the different binaries.

@wileyj
Copy link
Contributor

wileyj commented Jan 25, 2024

this may be a little tricky, and i have not tried something like this with rust projects so i'm not sure how much we can shrink the binaries.
there's a few standard methods we can try to shrink binaries, but there may also be some changes that would have to be made in how we compile, i.e. the cargo.toml file.

@wileyj
Copy link
Contributor

wileyj commented Jan 25, 2024

@BowTiedDevOps
Copy link
Collaborator

I built the binaries locally and stripped them afterwards, resulting in a significant size decrease.

Command ran: strip --strip-debug <bin>

OS: Debian 13

Size reduction:

  • stacks-node: from 358MB to 39MB
  • stacks-signer: from 208MB to 11MB

I've also tested the binaries after the strip to ensure that they don't panic and still work, but further testing must be done in order to be sure that they work as expected.

@wileyj
Copy link
Contributor

wileyj commented Jan 26, 2024

outstanding, thanks! we'll want to run the binaries though a bit more testing, but this looks very promising and simple change to make

@BowTiedDevOps
Copy link
Collaborator

Total size of all packages before reductions: 1545 MB

It looks like stripping the debug-info will be default in a future (1.77) stable release of cargo (currently implemented in nightly): Discussion comment

I found another way to strip, by also stripping all symbols (running just strip <bin>) instead debug-info only. This way of stripping removes too much from the binary, as an example running this command on stacks-signer removes the binary completely. These are the results:

  unstripped strip debug-info strip all symbols
stacks-node 374 MB 40 MB 34 MB
stacks-signer 363 MB 12 MB Removes bin file

By adding strip = "debuginfo" to [profile.release] inside the root Cargo.toml, I’ve gotten the total size of all the binaries to 94.691 MB, however, this strips the debug-info symbols from all the binaries, which may not be desired. In order to only strip specific binaries, you can run strip --strip-debug <bin>...<bin> after the release build, or add package.<package_name>.strip = "debuginfo" to [profile.release].

Size of binaries after the strip:

Using opt-level = "z" for the release profile (along with stripping) gives roughly the same result as simply stripping (stacks-node from 40 MB to 37 MB, stacks-signer - same size). The build time is decreased from 3m 33s to 2m 05s.

opt-level = "s" reduces the binaries’ size by about 1 MB more than opt-level = "z" (stacks-node from 40 MB → 36 MB, stacks-signer from 12 MB to 11 MB). The build time was increased from 3m 33s to 4m 06s.

Enabling link time optimization (lto = true in release profile) makes stackslib compile fail. However, building only specific binaries, stacks-node (6m 03s build) got to 36 MB and stacks-signer (2m 22s build) to 9.3 MB. Combining lto with opt-level = "s" decreases stacks-node to 30 MB and stacks-signer to 7.5 MB.

Reducing the parallel code generation units to 1 (codegen-units = 1) decreases the size of stacks-node to 28 MB and stacks-signer to 7.1 MB.

Aborting instead of unwinding on panic (panic = "abort") decreases stacks-node to 27 MB and stacks-signer to 6.5 MB.

Additions to the root Cargo.toml:

[profile.release]
debug = true
# package.stacks-node.strip = "debuginfo"
# package.stacks-signer.strip = "debuginfo"
strip = "debuginfo"
lto = true
opt-level = "s"
codegen-units = 1
panic = "abort"

It is also possible to parse these settings through the RUSTFLAGS environment variable, so you don’t have to add them inside the Cargo.toml file:

RUSTFLAGS="-C strip=debuginfo -C opt-level=s -C codegen-units=1 -C panic=abort" cargo build --features monitoring_prom,slog_json --release --workspace

Final size (with lto):

  • stacks-node - 27 MB
  • stacks-signer - 6.5 MB

However, having lto = "true" fails the build on stackslib, so, building without it gives us a final size of:

  • stacks-node - 30 MB
  • stacks-signer - 8.3 MB
  • All binaries - 67.178 MB

Using UPX tool in order to reduce the size of the binaries (upx --best --lzma <bin>...<bin>):

  • stacks-node - 15 MB
  • stacks-signer - 2.9 MB
  • All binaries - 27.612 MB

Size of binaries after all steps (without lto)

To sum it up, I've managed to reduce the total size of all the binaries from 1.545 GB to 67.178 MB (without UPX), or 27.612 MB with the binary compressor. I tested that the binaries still run, that the RUST_LOG_DEBUG=1 env still prints the debug messages, I've started a mock miner node successfully, tested that the stacks-signer get-chunk command works and blockstack-cli decode-tx successfully decodes a raw transaction. Every setting can be passed as env in CLI, so no files have to be modified, and the only thing that seems to be missing is the panic backtrace (which I removed with panic=abort, and it can be included back).

@hstove
Copy link
Contributor Author

hstove commented Jan 29, 2024

Is the main downside of stripping debug symbols that you can't get backtraces with RUST_BACKTRACE=1 ...?

@wileyj
Copy link
Contributor

wileyj commented Jan 30, 2024 via email

@wileyj
Copy link
Contributor

wileyj commented Jan 31, 2024

one more wrinkle that i've enountered here - it's only possible to add a release binary today for the source arch.
because of how stacks-signer currently builds, every other build fails (i.e. anything cross-compiled).
looking into options here, but it's definitely a blocker

@saralab saralab moved this from Status: 🆕 New to Status: 📋 Backlog in Stacks Core Eng Feb 13, 2024
@saralab saralab moved this from Status: 📋 Backlog to Status: 💻 In Progress in Stacks Core Eng Feb 26, 2024
@saralab saralab removed the 3.1 label Jun 21, 2024
@saralab saralab assigned BowTiedDevOps and unassigned wileyj Jun 25, 2024
@saralab saralab closed this as completed Oct 24, 2024
@github-project-automation github-project-automation bot moved this from Status: 💻 In Progress to Status: ✅ Done in Stacks Core Eng Oct 24, 2024
@blockstack-devops
Copy link
Contributor

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@stacks-network stacks-network locked as resolved and limited conversation to collaborators Nov 1, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
Status: Status: ✅ Done
Development

No branches or pull requests

5 participants