Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Link error with new openssl 1.1.0e #45

Closed
zorael opened this issue Apr 28, 2017 · 17 comments
Closed

Link error with new openssl 1.1.0e #45

zorael opened this issue Apr 28, 2017 · 17 comments

Comments

@zorael
Copy link
Contributor

zorael commented Apr 28, 2017

Arch linux recently updated their openssl package to 1.1.0e, and with it dlang-requests no longer links.

$ dub build
Performing "debug" build using dmd for x86_64.
requests 0.4.0: target for configuration "std" is up to date.
kameloso ~master: building configuration "kameloso"...
source/kameloso/config.d(66,13): Deprecation: 'switch' skips declaration of variable kameloso.config.walkConfigLines!(Result, IrcBot, IrcServer).walkConfigLines.thing at source/kameloso/config.d(68,13)
Linking...
../../.dub/packages/requests-0.4.0/requests/.dub/build/std-debug-linux.posix-x86_64-dmd_2074-E4CF0CB26A60CA71D7E3266518AEBC37/librequests.a(streams.o): In function `_D8requests7streams19_sharedStaticCtor46FZv':
/home/zorael/src/kameloso/../../.dub/packages/requests-0.4.0/requests/source/requests/streams.d:778: undefined reference to `SSL_library_init'
/home/zorael/src/kameloso/../../.dub/packages/requests-0.4.0/requests/source/requests/streams.d:779: undefined reference to `OpenSSL_add_all_ciphers'
/home/zorael/src/kameloso/../../.dub/packages/requests-0.4.0/requests/source/requests/streams.d:780: undefined reference to `OpenSSL_add_all_digests'
/home/zorael/src/kameloso/../../.dub/packages/requests-0.4.0/requests/source/requests/streams.d:781: undefined reference to `SSL_load_error_strings'
collect2: error: ld returned 1 exit status
Error: linker exited with status 1
dmd failed with exit code 1.

There now exists an openssl-1.0 package with the old library (.so-versioned), but installing it doesn't seem to be enough.

@ikod
Copy link
Owner

ikod commented Apr 28, 2017

Hello,
I have no arch linux under hands, need to setup vitrual machine to reproduce.

@ikod
Copy link
Owner

ikod commented Apr 29, 2017

Hello @zorael
I can reproduce problem. This is really compatibility problem between libssl1.0 and libssl1.1.
Workaround is to install openssl-1.0 and link libssl.so and crypto.so to 1.0 version
ln -s libssl.so.1.0.0 /usr/lib/libssl.so
ln -s libcrypto.so.1.0.0 /usr/lib/libcrypto.so
I'll check if it is possible to link requests with openssl.so.1.1 later.

@CounterPillow
Copy link

Downgrading is not really a solution. Ideally, Requests would be compatible with the OpenSSL 1.1 API, as it will need to do that sooner or later anyway.

From the OpenSSL documentation:

There are two ways to initialize the OpenSSL library, and they depend on the version of the library you are using. If you are using OpenSSL 1.0.2 or below, then you would use SSL_library_init. If you are using OpenSSL 1.1.0 or above, then you would use OPENSSL_init_ssl. A compatibility macro exists in ssl.h that maps SSL_library_init to OPENSSL_init_ssl, so you can continue to use SSL_library_init if desired. Also see SSL_library_init on the OpenSSL-dev mailing list.

The other API changes seem fairly easy to wrap in a compatible way too.

@CounterPillow
Copy link

Telling dlang-requests to link to the old libraries explicitly does work:

diff --git a/dub.json b/dub.json
index 3ae8ef3..4d6e259 100644
--- a/dub.json
+++ b/dub.json
@@ -9,7 +9,7 @@
     "configurations": [
         {
             "name": "std",
-            "libs-posix": ["ssl", "crypto"],
+            "libs-posix": [":libssl.so.1.0.0", ":libcrypto.so.1.0.0"],
             "libs-windows": ["libssl32", "libeay32"]
         },
         {

@pbackus
Copy link

pbackus commented Jul 9, 2017

This also affects Debian 9 (stable).

@ikod
Copy link
Owner

ikod commented Jul 9, 2017

Ok, will rise priority for this problem.
Does anybody know how to detect ssl version from inside the application (I have no access to ssl C headers to detect version at compile time)

@CounterPillow
Copy link

objdump -x shows that the shared library object does contain version information in the headers, so it should be possible if these headers get exposed to your code in some way.

@wilzbach
Copy link

wilzbach commented Jul 9, 2017

Does anybody know how to detect ssl version from inside the application (I have no access to ssl C headers to detect version at compile time)

A hack would be to use preBuildCommands, detect the current openssl version and then pass it as version (e.g. OPENSSL_1_1), though that's really ugly.

@CounterPillow
Copy link

Looks like OpenSSL does define symbols for each version, so you can check whether you can find a symbol called OPENSSL_1_1_0

@ikod
Copy link
Owner

ikod commented Jul 12, 2017

Hello,

Current plan is to try to use preBuildCommand to detect libssl version and then use it somehow to manage version for compilation.

I spent several days looking for portable way (targets are Linux, OSX, Windows, FreeBSD) to find current version of openssl library. I would like to avoid dependencies outside of D as much as possible - that is I'd like to avoid calling command line tools like 'openssl version -a', nm, objdump, readelf, etc, and compiling C sources.

To find which symbols are available we can use dlopen("libssl.so") and it will work for all unix-like OS's, but I do not know anything about Windows.

@CounterPillow @wilzbach what do you think?

@wilzbach
Copy link

@CounterPillow @wilzbach what do you think?

Sounds good to me. TBH I only really care about Linux ;-)

but I do not know anything about Windows.

Same here, maybe @CyberShadow knows a good solution for Windows?

Btw maybe the DerelictLoader SharedLibLoader will work?

@CyberShadow
Copy link

Sorry, my use of dub is entirely reluctant to ease use of my code by people who do use it, don't think I can help much with that.

As for dlopen, Windows' equivalent is LoadLibrary and GetProcAddress, that allows to probe which symbols are available in a DLL.

@bubnenkoff
Copy link

The is opinion that OpenSSL is suxx http://queue.acm.org/detail.cfm?id=2602816 and error prone.

@CounterPillow
Copy link

@bubnenkoff please keep discussion here constructive

@ikod
Copy link
Owner

ikod commented Jul 22, 2017

Hello,
This is just short note that work is going on.
Finally I tested portable way to initialise SSL for several available OSes and for both openssl 1.0 and 1.1.
Next plan - to map required openssl API calls to functions which will further use 1.0 or 1.1 dynamically loaded API calls to execute real job.

Right now it looks like

import std.stdio;
import std.string;
import std.format;
import std.typecons;
import core.stdc.stdlib;
import core.sys.posix.dlfcn;

version(Windows) {
    import core.sys.windows.windows;
    alias DLSYM = GetProcAddress;
} else {
    alias DLSYM = dlsym;
}

static OpenSSL openssl;

static this() {
    openssl.loadLib();
    openssl.initSSL();
}

struct OpenSSL {
    alias Version = Tuple!(int, "major", int, "minor");
    Version         _ver;
    void*           _libssl;
    void*           _libcrypto;

    Version reportVersion() const @nogc nothrow pure {
        return _ver;
    };

    void OpenSSL_version_detect() {
        long function() OpenSSL_version_num = cast(long function())DLSYM(_libssl, "OpenSSL_version_num");
        if ( OpenSSL_version_num ) {
            auto v = OpenSSL_version_num();
            _ver = Version((v>>>20) & 0xff, (v>>>28) & 0xff);
            return;
        }
        _ver = Version(1, 0);
    }

    void init1_0() {
        extern (C) int function() SSL_library_init = cast(int function())DLSYM(_libssl, "SSL_library_init");
        extern (C) void function() OpenSSL_add_all_ciphers = cast(void function())DLSYM(_libcrypto, "OpenSSL_add_all_ciphers");
        extern (C) void function() OpenSSL_add_all_digests = cast(void function())DLSYM(_libcrypto, "OpenSSL_add_all_digests");
        extern (C) void function() SSL_load_error_strings = cast(void function())DLSYM(_libssl, "SSL_load_error_strings");
        SSL_library_init();
        OpenSSL_add_all_ciphers();
        OpenSSL_add_all_digests();
        SSL_load_error_strings();
    }
    void init1_1() {
        /**
        Standard initialisation options

        #define OPENSSL_INIT_LOAD_SSL_STRINGS       0x00200000L

        # define OPENSSL_INIT_LOAD_CRYPTO_STRINGS    0x00000002L
        # define OPENSSL_INIT_ADD_ALL_CIPHERS        0x00000004L
        # define OPENSSL_INIT_ADD_ALL_DIGESTS        0x00000008L
        **/
        enum OPENSSL_INIT_LOAD_SSL_STRINGS = 0x00200000L;
        enum OPENSSL_INIT_LOAD_CRYPTO_STRINGS = 0x00000002L;
        enum OPENSSL_INIT_ADD_ALL_CIPHERS = 0x00000004L;
        enum OPENSSL_INIT_ADD_ALL_DIGESTS = 0x00000008L;
        extern (C) int function(ulong, void*) OPENSSL_init_ssl = cast(int function(ulong, void*))DLSYM(_libssl, "OPENSSL_init_ssl");
        extern (C) int function(ulong, void*) OPENSSL_init_crypto = cast(int function(ulong, void*))DLSYM(_libcrypto, "OPENSSL_init_crypto");
        OPENSSL_init_ssl(OPENSSL_INIT_LOAD_SSL_STRINGS, null);
        OPENSSL_init_crypto(OPENSSL_INIT_ADD_ALL_CIPHERS | OPENSSL_INIT_ADD_ALL_DIGESTS | OPENSSL_INIT_LOAD_CRYPTO_STRINGS, null);
    }
    void initSSL() {
        void delegate()[Version] init_matrix;
        init_matrix[Version(1,0)] = &init1_0;
        init_matrix[Version(1,1)] = &init1_1;
        auto init = init_matrix.get(_ver, null);
        if ( init is null ) {
            throw new Exception("loading openssl: unknown version for init");
        }
        init();
    }

    void loadLib() {
        if ( _libssl !is null ) {
            return;
        }
        version(OSX) {
            _libssl = dlopen("libssl.dylib", RTLD_LAZY);
            _libcrypto = dlopen("libcrypto.dylib", RTLD_LAZY);
        } else
        version(linux) {
            _libssl = dlopen("libssl.so", RTLD_LAZY);
            _libcrypto = dlopen("libcrypto.so", RTLD_LAZY);
        } else
        version(Windows) {
            _libssl = LoadLibrary("libssl32.dll");
            _libcrypto = LoadLibrary("libeay32.dll");
        } else {
            throw new Exception("loading openssl: unsupported system");
        }
        if ( _libssl is null ) {
            version(Windows) {
                throw new Exception("loading libssl: error %d".format(GetLastError()));
            } else {
                throw new Exception("loading openssl: %s",format(fromStringz(dlerror())));
            }
        }
        if ( _libcrypto is null ) {
            version(Windows) {
                throw new Exception("loading libcrypto: error %d".format(GetLastError()));
            } else {
                throw new Exception("loading libcrypto: %s",format(fromStringz(dlerror())));
            }
        }
        OpenSSL_version_detect();
    }
}
int main() {
    auto v = openssl.reportVersion();
    writefln("openSSL v.%s.%s", v.major, v.minor);
  return 0;
}

@ikod
Copy link
Owner

ikod commented Aug 1, 2017

Hello,

I pushed to github what was done for openssl 1.0/1.1 linking. Code tested for openssl 1.0 on Linux, OSX, Windows7, and openssl 1.1 tested on debian "stretch".

Please, check and report on any issues.

Thanks!

PS. I will make new release in few days, so anybody can try it.

@ikod
Copy link
Owner

ikod commented Aug 3, 2017

Hello, @zorael
I will close this issue as it should be solved in 0.5.0 release.
Please, reopen in case of any problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants