Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Patch out any instances of printf in upstream #663

Merged
merged 3 commits into from
Nov 15, 2023

Conversation

apoelstra
Copy link
Member

Rather than using a new patchfile, just #define it away. Also includes a commit which removes one of the existing patchfiles, which I discovered was out of date while auditing the others to see if they could be replaced by #defines. (No, they cannot.)

Fixes #660

This patch adds a declaration of the `ecdsa_parse_compact` function to
util.h. This function isn't called from within libsecp proper; it is
called in lax_der_parse.c (which we patch separately with a declaration)
and in example code (which we don't compile at all).
Copy link
Collaborator

@Kixunil Kixunil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 942a0e5

@@ -20,11 +20,16 @@ fn main() {
.include("depend/secp256k1/include")
.include("depend/secp256k1/src")
.flag_if_supported("-Wno-unused-function") // some ecmult stuff is defined but not used upstream
.flag_if_supported("-Wno-unused-parameter") // patching out printf causes this warning
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this can be solved more locally but I haven't used C for a while so I don't remember how.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no standard C way to do it. Any specific compiler would have annotations but they wouldn't work for others. (Of course, flag_if_supported also not universal. But cc makes an effort to translate it for the common compilers.)

And in any case, if I wanted a more local solution, I'd just patch out the offending methods entirely :).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean something like using (void)x works IIRC which can be used in the define. But I don't know how to apply it to multiple arguments.

@apoelstra
Copy link
Member Author

@cryptoquick are you able to test this PR (or just its last commit) and see if it works for you?

@cryptoquick
Copy link

When I add these crates:

secp256k1 = { git = "https://github.com/apoelstra/rust-secp256k1", branch = "2023-11--define-not-patch" }
secp256k1-sys = { git = "https://github.com/apoelstra/rust-secp256k1", branch = "2023-11--define-not-patch" }

I get this error:

[INFO]: 🎯  Checking for the Wasm target...
[INFO]: 🌀  Compiling to Wasm...
   Compiling secp256k1-sys v0.9.0 (https://github.com/apoelstra/rust-secp256k1?branch=2023-11--define-not-patch#942a0e5e)
The following warnings were emitted during compilation:

warning: In file included from depend/secp256k1/src/precomputed_ecmult_gen.c:3:
warning: In file included from depend/secp256k1/src/group.h:10:
warning: In file included from depend/secp256k1/src/field.h:10:
warning: depend/secp256k1/src/util.h:28:50: warning: unused parameter 'buf' [-Wunused-parameter]
warning: static void print_buf_plain(const unsigned char *buf, size_t len) {
warning:                                                  ^
warning: depend/secp256k1/src/util.h:148:17: error: call to undeclared library function 'malloc' with type 'void *(unsigned long)'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
warning:     void *ret = malloc(size);
warning:                 ^
warning: depend/secp256k1/src/util.h:148:17: note: include the header <stdlib.h> or explicitly provide a declaration for 'malloc'
warning: 1 warning and 1 error generated.
warning: In file included from depend/secp256k1/src/precomputed_ecmult_gen.c:3:
warning: In file included from depend/secp256k1/src/group.h:10:
warning: In file included from depend/secp256k1/src/field.h:10:
warning: depend/secp256k1/src/util.h:28:50: warning: unused parameter 'buf' [-Wunused-parameter]
warning: static void print_buf_plain(const unsigned char *buf, size_t len) {
warning:                                                  ^
warning: depend/secp256k1/src/util.h:148:17: error: call to undeclared library function 'malloc' with type 'void *(unsigned long)'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
warning:     void *ret = malloc(size);
warning:                 ^
warning: depend/secp256k1/src/util.h:148:17: note: include the header <stdlib.h> or explicitly provide a declaration for 'malloc'
warning: 1 warning and 1 error generated.

error: failed to run custom build command for `secp256k1-sys v0.9.0 (https://github.com/apoelstra/rust-secp256k1?branch=2023-11--define-not-patch#942a0e5e)`

Caused by:
  process didn't exit successfully: `/home/hunter/Projects/diba/bitmask-core-5/target/release/build/secp256k1-sys-2782e3689c5032f0/build-script-build` (exit status: 1)
  --- stdout
  TARGET = Some("wasm32-unknown-unknown")
  OPT_LEVEL = Some("3")
  HOST = Some("x86_64-unknown-linux-gnu")
  cargo:rerun-if-env-changed=CC_wasm32-unknown-unknown
  CC_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CC_wasm32_unknown_unknown
  CC_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")
  cargo:rerun-if-env-changed=CFLAGS_wasm32-unknown-unknown
  CFLAGS_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32_unknown_unknown
  CFLAGS_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CC_wasm32-unknown-unknown
  CC_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CC_wasm32_unknown_unknown
  CC_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32-unknown-unknown
  CFLAGS_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32_unknown_unknown
  CFLAGS_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CC_wasm32-unknown-unknown
  CC_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CC_wasm32_unknown_unknown
  CC_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32-unknown-unknown
  CFLAGS_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32_unknown_unknown
  CFLAGS_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  running: "clang" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=wasm32-unknown-unknown" "-I" "depend/secp256k1/" "-I" "depend/secp256k1/include" "-I" "depend/secp256k1/src" "-I" "wasm/wasm-sysroot" "-Wall" "-Wextra" "-DSECP256K1_API=" "-DENABLE_MODULE_ECDH=1" "-DENABLE_MODULE_SCHNORRSIG=1" "-DENABLE_MODULE_EXTRAKEYS=1" "-DENABLE_MODULE_ELLSWIFT=1" "-Dprintf(...)=" "-DECMULT_GEN_PREC_BITS=4" "-DECMULT_WINDOW_SIZE=15" "-DUSE_EXTERNAL_DEFAULT_CALLBACKS=1" "-o" "/home/hunter/Projects/diba/bitmask-core-5/target/wasm32-unknown-unknown/release/build/secp256k1-sys-750d394d56fd67d3/out/wasm/wasm.o" "-c" "wasm/wasm.c"
  exit status: 0
  running: "clang" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=wasm32-unknown-unknown" "-I" "depend/secp256k1/" "-I" "depend/secp256k1/include" "-I" "depend/secp256k1/src" "-I" "wasm/wasm-sysroot" "-Wall" "-Wextra" "-DSECP256K1_API=" "-DENABLE_MODULE_ECDH=1" "-DENABLE_MODULE_SCHNORRSIG=1" "-DENABLE_MODULE_EXTRAKEYS=1" "-DENABLE_MODULE_ELLSWIFT=1" "-Dprintf(...)=" "-DECMULT_GEN_PREC_BITS=4" "-DECMULT_WINDOW_SIZE=15" "-DUSE_EXTERNAL_DEFAULT_CALLBACKS=1" "-o" "/home/hunter/Projects/diba/bitmask-core-5/target/wasm32-unknown-unknown/release/build/secp256k1-sys-750d394d56fd67d3/out/depend/secp256k1/contrib/lax_der_parsing.o" "-c" "depend/secp256k1/contrib/lax_der_parsing.c"
  exit status: 0
  running: "clang" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=wasm32-unknown-unknown" "-I" "depend/secp256k1/" "-I" "depend/secp256k1/include" "-I" "depend/secp256k1/src" "-I" "wasm/wasm-sysroot" "-Wall" "-Wextra" "-DSECP256K1_API=" "-DENABLE_MODULE_ECDH=1" "-DENABLE_MODULE_SCHNORRSIG=1" "-DENABLE_MODULE_EXTRAKEYS=1" "-DENABLE_MODULE_ELLSWIFT=1" "-Dprintf(...)=" "-DECMULT_GEN_PREC_BITS=4" "-DECMULT_WINDOW_SIZE=15" "-DUSE_EXTERNAL_DEFAULT_CALLBACKS=1" "-o" "/home/hunter/Projects/diba/bitmask-core-5/target/wasm32-unknown-unknown/release/build/secp256k1-sys-750d394d56fd67d3/out/depend/secp256k1/src/precomputed_ecmult_gen.o" "-c" "depend/secp256k1/src/precomputed_ecmult_gen.c"
  cargo:warning=In file included from depend/secp256k1/src/precomputed_ecmult_gen.c:3:

  cargo:warning=In file included from depend/secp256k1/src/group.h:10:

  cargo:warning=In file included from depend/secp256k1/src/field.h:10:

  cargo:warning=depend/secp256k1/src/util.h:28:50: warning: unused parameter 'buf' [-Wunused-parameter]

  cargo:warning=static void print_buf_plain(const unsigned char *buf, size_t len) {

  cargo:warning=                                                 ^

  cargo:warning=depend/secp256k1/src/util.h:148:17: error: call to undeclared library function 'malloc' with type 'void *(unsigned long)'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]

  cargo:warning=    void *ret = malloc(size);

  cargo:warning=                ^

  cargo:warning=depend/secp256k1/src/util.h:148:17: note: include the header <stdlib.h> or explicitly provide a declaration for 'malloc'

  cargo:warning=1 warning and 1 error generated.

  exit status: 1
  running: "clang" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=wasm32-unknown-unknown" "-I" "depend/secp256k1/" "-I" "depend/secp256k1/include" "-I" "depend/secp256k1/src" "-I" "wasm/wasm-sysroot" "-I" "wasm/wasm-sysroot" "-Wall" "-Wextra" "-DSECP256K1_API=" "-DENABLE_MODULE_ECDH=1" "-DENABLE_MODULE_SCHNORRSIG=1" "-DENABLE_MODULE_EXTRAKEYS=1" "-DENABLE_MODULE_ELLSWIFT=1" "-Dprintf(...)=" "-DECMULT_GEN_PREC_BITS=4" "-DECMULT_WINDOW_SIZE=15" "-DUSE_EXTERNAL_DEFAULT_CALLBACKS=1" "-o" "/home/hunter/Projects/diba/bitmask-core-5/target/wasm32-unknown-unknown/release/build/secp256k1-sys-750d394d56fd67d3/out/wasm/wasm.o" "-c" "wasm/wasm.c"
  exit status: 0
  running: "clang" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=wasm32-unknown-unknown" "-I" "depend/secp256k1/" "-I" "depend/secp256k1/include" "-I" "depend/secp256k1/src" "-I" "wasm/wasm-sysroot" "-I" "wasm/wasm-sysroot" "-Wall" "-Wextra" "-DSECP256K1_API=" "-DENABLE_MODULE_ECDH=1" "-DENABLE_MODULE_SCHNORRSIG=1" "-DENABLE_MODULE_EXTRAKEYS=1" "-DENABLE_MODULE_ELLSWIFT=1" "-Dprintf(...)=" "-DECMULT_GEN_PREC_BITS=4" "-DECMULT_WINDOW_SIZE=15" "-DUSE_EXTERNAL_DEFAULT_CALLBACKS=1" "-o" "/home/hunter/Projects/diba/bitmask-core-5/target/wasm32-unknown-unknown/release/build/secp256k1-sys-750d394d56fd67d3/out/depend/secp256k1/contrib/lax_der_parsing.o" "-c" "depend/secp256k1/contrib/lax_der_parsing.c"
  exit status: 0
  running: "clang" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=wasm32-unknown-unknown" "-I" "depend/secp256k1/" "-I" "depend/secp256k1/include" "-I" "depend/secp256k1/src" "-I" "wasm/wasm-sysroot" "-I" "wasm/wasm-sysroot" "-Wall" "-Wextra" "-DSECP256K1_API=" "-DENABLE_MODULE_ECDH=1" "-DENABLE_MODULE_SCHNORRSIG=1" "-DENABLE_MODULE_EXTRAKEYS=1" "-DENABLE_MODULE_ELLSWIFT=1" "-Dprintf(...)=" "-DECMULT_GEN_PREC_BITS=4" "-DECMULT_WINDOW_SIZE=15" "-DUSE_EXTERNAL_DEFAULT_CALLBACKS=1" "-o" "/home/hunter/Projects/diba/bitmask-core-5/target/wasm32-unknown-unknown/release/build/secp256k1-sys-750d394d56fd67d3/out/depend/secp256k1/src/precomputed_ecmult_gen.o" "-c" "depend/secp256k1/src/precomputed_ecmult_gen.c"
  cargo:warning=In file included from depend/secp256k1/src/precomputed_ecmult_gen.c:3:

  cargo:warning=In file included from depend/secp256k1/src/group.h:10:

  cargo:warning=In file included from depend/secp256k1/src/field.h:10:

  cargo:warning=depend/secp256k1/src/util.h:28:50: warning: unused parameter 'buf' [-Wunused-parameter]

  cargo:warning=static void print_buf_plain(const unsigned char *buf, size_t len) {

  cargo:warning=                                                 ^

  cargo:warning=depend/secp256k1/src/util.h:148:17: error: call to undeclared library function 'malloc' with type 'void *(unsigned long)'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]

  cargo:warning=    void *ret = malloc(size);

  cargo:warning=                ^

  cargo:warning=depend/secp256k1/src/util.h:148:17: note: include the header <stdlib.h> or explicitly provide a declaration for 'malloc'

  cargo:warning=1 warning and 1 error generated.

  exit status: 1

  --- stderr


  error occurred: Command "clang" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=wasm32-unknown-unknown" "-I" "depend/secp256k1/" "-I" "depend/secp256k1/include" "-I" "depend/secp256k1/src" "-I" "wasm/wasm-sysroot" "-I" "wasm/wasm-sysroot" "-Wall" "-Wextra" "-DSECP256K1_API=" "-DENABLE_MODULE_ECDH=1" "-DENABLE_MODULE_SCHNORRSIG=1" "-DENABLE_MODULE_EXTRAKEYS=1" "-DENABLE_MODULE_ELLSWIFT=1" "-Dprintf(...)=" "-DECMULT_GEN_PREC_BITS=4" "-DECMULT_WINDOW_SIZE=15" "-DUSE_EXTERNAL_DEFAULT_CALLBACKS=1" "-o" "/home/hunter/Projects/diba/bitmask-core-5/target/wasm32-unknown-unknown/release/build/secp256k1-sys-750d394d56fd67d3/out/depend/secp256k1/src/precomputed_ecmult_gen.o" "-c" "depend/secp256k1/src/precomputed_ecmult_gen.c" with args "clang" did not execute successfully (status code exit status: 1).

Is this the proper way to test this PR?

@apoelstra
Copy link
Member Author

Thanks! Yes, it looks like you're testing it correctly.

Very strange that your compiler is getting upset at the use of malloc inside of checked_malloc, which isn't even called anywhere in the library. The strangeness is because this has been the case for years, but til now nobody has complained.

I will add to this PR to replace checked_malloc with a stub that unconditionally calls the error callback, which should address this issue. Though possibly there will be more errors to come.

@tcharding
Copy link
Member

tcharding commented Nov 2, 2023

Ok, I got rid of all warnings on my machine by using:

    // WASM headers and size/align defines.
    if env::var("CARGO_CFG_TARGET_ARCH").unwrap() == "wasm32" {
        base_config.include("wasm/wasm-sysroot")
            // upstream sometimes introduces calls to printf, which we cannot compile
            // with WASM due to its lack of libc. printf is never necessary and we can
            // just #define it away.
            .define("printf(...)", Some(""))
            .flag("-Wno-unused-parameter") // patching out printf causes this warning
	    .flag("-Wno-unused-function") // some ecmult stuff is defined but not used upstream      
            .flag("-Wno-implicit-function-declaration")
            .file("wasm/wasm.c");
    }

I believe its ok to use flag instead of flag_if_supported because we know wasm32 targets use clang and clang supports these flags.

// upstream sometimes introduces calls to printf, which we cannot compile
// with WASM due to its lack of libc. printf is never necessary and we can
// just #define it away.
.define("printf(...)", Some(""));
Copy link
Member

@tcharding tcharding Nov 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should only use this if the target is wasm ie., put it in:

    // WASM headers and size/align defines.
    if env::var("CARGO_CFG_TARGET_ARCH").unwrap() == "wasm32" {

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also the -Wno-unused-paramater

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because there is a bug in cc::Build::flag_if_supported so we are getting loads of warnings. Well on my machine anyways.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why?

Did you mean why do I think we should put the .define on the wasm-only cc::Build or why the "Also the -Wno-unused-parameter? Ignore the second one, and for the first, I thought this because the code comment specifically mentions that this is a wasm problem so it seems logical to put it in the wasm-only place.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's wasm-only. It would be any linker that tries to resolve libc symbols before pruning unused code.

@apoelstra
Copy link
Member Author

I believe its ok to use flag instead of flag_if_supported because we know wasm32 targets use clang and clang supports these flags.

Ok, but what about every other target we support? Do we know that they will use gcc or clang?

@cryptoquick
Copy link

I'm not sure gcc figures into wasm compilation. I'm using clang. Here are the details on my system:
#660 (comment)

@tcharding
Copy link
Member

Here is the solution to the wasm/flag_if_supported thing: rust-lang/cc-rs#839

@Kixunil
Copy link
Collaborator

Kixunil commented Nov 5, 2023

@tcharding so looks like @cryptoquick just needs to run cargo update -p cc?

@cryptoquick
Copy link

cryptoquick commented Nov 5, 2023

@Kixunil Tried that, but I'm already running [email protected]. I noticed there was an update to libc v0.2.150, from 149, but that did not help.

Just realized, the fix that was merged has not yet been released. Will have to keep an eye on that.

@Kixunil
Copy link
Collaborator

Kixunil commented Nov 6, 2023

the fix that was merged has not yet been released

Oh, didn't notice. Perhaps test with their master branch using [patch]?

@apoelstra
Copy link
Member Author

I will add to this PR to replace checked_malloc with a stub that unconditionally calls the error callback, which should address this issue. Though possibly there will be more errors to come.

Oops, just did this now.

Was not as simple as using a #define since the function is actually defined, and trying to stub it out with the preprocessor messes up the definition.

@@ -145,11 +145,9 @@ static const rustsecp256k1_v0_9_0_callback default_error_callback = {
#endif

static SECP256K1_INLINE void *checked_malloc(const rustsecp256k1_v0_9_0_callback* cb, size_t size) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe delete it completely to ensure it's not called by accident?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then I would need to delete all the functions that call it, and so on, which is much more invasive.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sucks. Also I wonder how does it interact with the system library. Will anything break if it has different code?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. It's not an exposed function, it's only used internally.

@tcharding
Copy link
Member

AFAICT this is right to go, ACK 7a0c60e

Copy link
Collaborator

@Kixunil Kixunil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 7a0c60e

@apoelstra apoelstra merged commit 023d50b into rust-bitcoin:master Nov 15, 2023
20 checks passed
@apoelstra apoelstra deleted the 2023-11--define-not-patch branch November 15, 2023 18:28
@apoelstra
Copy link
Member Author

Heh, github warned me that I'm bypassing the "one ack from a maintainer" rule. But I think that something I authored, with 2 acks from rust-bitcoin maintainers, and the original bug reporter saying that it helped, is ok to merge.

@cryptoquick
Copy link

Just tried the master branch with wasm-pack, it works great, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Consistently getting compilation error when compiling to wasm32
4 participants