Support for CROSSTOOL with custom dynamic runtime libs w/o using EXEC_ORIGIN

[jfroy]

Our project targets a slightly strange flavor of Linux where the runtime environment does not use libstdc++. We built a custom CROSSTOOL with clang/llvm and a minimal sysroot, and we can compile programs, but we cannot run tests because the sysroot runtime libraries are not made available to test programs. Using dynamic_runtime_libs in cc_toolchain and supports_embedded_runtimes: true in CROSSTOOL does not work because the standard Linux dynamic linker doesn't support EXEC_ORIGIN (at least on the Debian and Ubuntu workstations we use). We don't want to use a pile of hacks to solve this, so we'd like to see a solution in bazel for this use case.

Environment info

• Operating System:
  Ubuntu 14.04 LTS
  Debian unstable

• Bazel version (output of bazel info release):
  release 0.5.3

[hlopko]

Hi Jean, sorry for the silence. Is the only problem that bazel emits $EXEC_ORIGIN instead of $ORIGIN with your crosstool?

[jfroy]

It's not clear at all to us how we can craft a CROSSTOOL that will allow, say, bazel test to work without having to specify our runtime libraries as a data dependency. That's our current workaround, using custom cc rules for all our targets. It's not ideal, since we now have custom rules implicitly adding a dependency on a specific crosstool.

Our CROSSTOOL's BUILD file looks something like this:

package(default_visibility = ["//visibility:public"])

cc_toolchain_suite(
    name = "toolchain",
    toolchains = {
        "k8|clang": ":cc-compiler-k8",
    },
)

cc_toolchain(
    name = "cc-compiler-k8",
    all_files = ":all_files-k8",
    compiler_files = ":all_files-k8",
    cpu = "k8",
    dwp_files = ":empty",
    dynamic_runtime_libs = [":dynamic-runtime-libs-k8"],
    linker_files = ":all_files-k8",
    objcopy_files = ":all_files-k8",
    static_runtime_libs = [":static-runtime-libs-k8"],
    strip_files = ":all_files-k8",
    supports_param_files = 1,
)

filegroup(
    name = "all_files-k8",
    srcs = [
        ":sysroot_files-k8",
        ":toolchain_files-k8",
    ],
)

filegroup(
    name = "toolchain_files-k8",
    srcs = glob(["toolchain/**/*"]),
)

filegroup(
    name = "sysroot_files-k8",
    srcs = glob(["sysroot/**/*"]),
)

filegroup(
    name = "static-runtime-libs-k8",
    srcs = glob(["sysroot/lib/*.a"]),
)

filegroup(
    name = "dynamic-runtime-libs-k8",
    srcs = glob(["sysroot/lib/*.so*"]),
)

filegroup(
    name = "empty",
    srcs = [],
)

The CROSSTOOL is generated from a template downloaded from Cloud Storage using a repository_rule in our WORKSPACE to have absolute paths, and looks like this:

major_version: "4"
minor_version: "0"

default_target_cpu: "k8"
default_toolchain {
  cpu: "k8"
  toolchain_identifier: "clang_k8"
}

toolchain {
  toolchain_identifier: "clang_k8"

  host_system_name: "x86_64-unknown-linux-gnu"
  target_system_name: "x86_64-unknown-linux-gnu"
  target_cpu: "k8"
  target_libc: "glibc_2.23"
  compiler: "clang"

  abi_version: "clang_5.0.1"
  abi_libc_version: "glibc_2.23"

  builtin_sysroot: "external/<external name>/sysroot"

  tool_path { name: "ar" path: "<abs path>/toolchain/bin/ar" }
  tool_path { name: "cpp" path: "<abs path>/toolchain/bin/clang++" }
  tool_path { name: "gcc" path: "<abs path>/toolchain/bin/clang" }
  tool_path { name: "gcov" path: "<abs path>/toolchain/bin/llvm-cov" }
  tool_path { name: "ld" path: "<abs path>/toolchain/bin/lld" }
  tool_path { name: "nm" path: "<abs path>/toolchain/bin/nm" }
  tool_path { name: "objcopy" path: "<abs path>/toolchain/bin/objcopy" }
  tool_path { name: "objdump" path: "<abs path>/toolchain/bin/objdump" }
  tool_path { name: "strip" path: "<abs path>/toolchain/bin/strip" }

  supports_gold_linker: true
  supports_thin_archives: true
  supports_start_end_lib: true
  supports_interface_shared_objects: true
  supports_embedded_runtimes: false
  supports_incremental_linker: true
  supports_normalizing_ar: true
  supports_fission: false
  needsPic: true

  cxx_builtin_include_directory: "<abs path>/toolchain/include/c++/v1/"
  cxx_builtin_include_directory: "<abs path>/toolchain/lib/clang/4.0.1/include"

  # The default linker is lld.
  linker_flag: "-fuse-ld=lld"

  # Link libc++ by default. The default cpp CROSSTOOL also does this.
  linker_flag: "-lc++"

  # Append the sysroot's lib directory to the rpath such that binaries will find
  # libc++ provided it is included as a data dependency when using bazel run.
  linker_flag: "-Wl,-rpath,external/<external name>/sysroot/lib"

<snip>
}

[hlopko]

I'm not sure I understand the problem, are you saying that you want to specify cc_toolchain without having to provide toolchain inputs via ":all_files-k8"?

One detail, I see you have supports_embedded_runtimes: false but yet you provide static_runtime_libs and dynamic_runtime_libs filegroups on cc_toolchain. If your toolchain doesn't support embedded runtimes these filegroups are ignored.

One more question, why do you need absolute paths in your crosstool if you provide the hermetic toolchain (i.e. you're not using the toolchain that is installed on the host system). In this case it's usual to use relative paths. I'm just curious.

Thanks!

[jfroy]

The problem is, with that CROSSTOOL, if I do

bazel test --crosstool_top=@mycrosstool//:toolchain //...

All the tests will fail because libc++ and libc++abi will not be in the sandbox.

If I set supports_embedded_runtimes: true, then bazel does copy them, but uses EXEC_ORIGIN, which causes any program to fail to launch on Debian and Ubuntu (my desktop is a Debian unstable derivative).

The absolute paths solve some strange problems we've had with cxx_builtin_include_directory and cgo. We don't hardcode them, we use a repository_rule to generate the final CROSSTOOL from a template.

[hlopko]

Heh maybe you encountered the same problem we're discussing with @jayconrod at this very moment over here: bazelbuild/rules_go#1357

If you put libc++ library into all_files-k8 it will be in the sandbox. Will that help?

[dapirian]

@mhlopko I replicated a similar setup as @jfroy described - basically with the goal of having a hermetic clang and libc++ pulled in with bazel via http_archive. I eventually got it working but it took adding this to my crosstool to make EXEC_ORIGIN just ORIGIN because ubuntu 14.04 and 16.04 don't support EXEC_ORIGIN...

feature {
        name: 'runtime_library_search_directories',
        flag_set {
            expand_if_all_available: 'runtime_library_search_directories'
            action: 'c++-link-executable'
            action: 'c++-link-dynamic-library'
            action: 'c++-link-nodeps-dynamic-library'
            flag_group {
                iterate_over: 'runtime_library_search_directories'
                flag_group {
                    flag: '-Wl,-rpath,$ORIGIN/%{runtime_library_search_directories}'
                }
            }
        }
    }

ORIGIN works, I don't know why EXEC_ORIGIN was used there, but it's very un-supported, can it be changed to ORIGIN on master?

[jfroy]

@dapirian Thanks, I will try that. Do you mind sharing your whole CROSSTOOL?

[dapirian]

@jfroy Here's my repo: https://github.com/dapirian/m
I'm thinking that as an example is going to help a lot of people. It pulls a prebuilt clang binary from llvm via http_archive and uses it (with clang, libc++, lld, and all the rest) hermetically, without terrible hacks to do so. I don't have code coverage working yet though (could use some help there... not sure what's wrong), and I'd like to use cc_configure instead of having a CROSSTOOL but I don't know enough to know what I want yet on that front.

[hlopko]

So removing EXEC_ORIGIN is pending a small internal migration, but not too hard. Then I can replace it with ORIGIN in CppActionConfigs. Until then the workaround is to define your own runtime_library_search_directories feature like @dapirian does. Thanks!

[blaizard]

Changing $EXEC_ORIGIN with $ORIGIN doesn't seems to fix the issue on my setup (Ubuntu 18 & Bazel 2.0.0). The shared libraries needed at runtime are located under the *.runfiles/ directory which is also where tests are executed. The test binary under this directory tree is a symlink. So I believe it is supposed to work with $EXEC_ORIGIN on a machine where the dynamic linker supports it (note the difference between $ORIGIN and $EXEC_ORIGIN is that the first one resolves from the actual targeted binary, while the latter one from the executed target, so a symlink in that case).
I see 2 possible ways of fixing this:

1. Do not add the shared libraries needed at runtime under *.runfiles/ but above where the actual binary is. This would make it work with $ORIGIN.
2. Add a bazel switch to copy (instead of symlink) the binary under *.runfiles/

Seems to be a similar concern in #6700

[blaizard]

As a workaround, you can actual use the following. Seems to works as Bazel run/test binaries from the root of the *.runfiles/ tree:
(notice the removal of $ORIGIN/$EXEC_ORIGIN)

feature {
        name: 'runtime_library_search_directories',
        flag_set {
            expand_if_all_available: 'runtime_library_search_directories'
            action: 'c++-link-executable'
            action: 'c++-link-dynamic-library'
            action: 'c++-link-nodeps-dynamic-library'
            flag_group {
                iterate_over: 'runtime_library_search_directories'
                flag_group {
                    flag: '-Wl,-rpath,%{runtime_library_search_directories}'
                }
            }
        }
    }

[carlosgalvezp]

Hi!
I'm interested solving the same problem - use a hermetic GCC toolchain and make sure the produced binaries will pick the correct libstdc++, libgcc, etc. instead of the system-wide ones at runtime.

The solution proposed by @blaizard seems exactly what I'm looking for. Just one caveat:

            expand_if_all_available: 'runtime_library_search_directories'

How do I populate runtime_library_search_directories? How do I make them available?

[carlosgalvezp]

@jfroy Here's my repo: https://github.com/dapirian/m I'm thinking that as an example is going to help a lot of people. It pulls a prebuilt clang binary from llvm via http_archive and uses it (with clang, libc++, lld, and all the rest) hermetically, without terrible hacks to do so. I don't have code coverage working yet though (could use some help there... not sure what's wrong), and I'd like to use cc_configure instead of having a CROSSTOOL but I don't know enough to know what I want yet on that front.

@dapirian I'm interested in solving exactly the same problem but the repo is no longer available. Do you mind sharing it? Thanks!

[ltekieli]

@carlosgalvezp I wanted also to understand this topic and did a small reproducer:

• https://github.com/ltekieli/bazel_hermetic_x86_64_toolchain
• https://github.com/ltekieli/bazel_hermetic_x86_64_toolchain/blob/5cbadaf3063c842705994ce16ebf2f75b7ffb643/bazel/toolchain/x86_64-unknown-linux-gnu/cc_toolchain_config.bzl#L168

This "dynamic_runtime_lib" thing seems to me a bit like a workaround. I cannot find the code responsible for controlling the 'runtime_library_search_directories' variable.

Btw, @hlopko does any actual example of a fully hermetic toolchain exist? I have a shameful workaround, but this definitely does not work with remote execution:

• https://github.com/ltekieli/bazel_hermetic_x86_64_toolchain/blob/5cbadaf3063c842705994ce16ebf2f75b7ffb643/bazel/toolchain/x86_64-unknown-linux-gnu/wrappers/wrapper#L24

[carlosgalvezp]

@ltekieli thanks, that's exactly what I was looking for! I did try the dynamic_runtime_lib bit, but I was missing to set linkstatic=false. Your README was very enlightening.

On a side note, I'm no longer sure this is the right way to go. If we depend on a third-party .so file that itself depends on libstdc++ (fairly common), then we might have issues due to the linker picking symbols from different libstdc++. The binary will use the hermetic libstdc++, but libthird_party.so will use symbols from the system libstdc++.

Therefore I believe the safest solution is to create a custom cc_binary rule that generates a wrapper around the actual binary. The wrapper should then set the correct LD_LIBRARY_PATH - this way all dependencies will use the same core libraries like libstdc++.

About the dynamic linker, it is indeed a problem. I found that the dynamic linker and glibc need to match, otherwise the binary crashes with segfault. So perhaps a middle-point solution is to use system-wide dynamic linker glibc, and use hermetic libstdc++, libgcc and so on?

[adrianimboden]

For future reference, I had to add this feature to the cc_toolchain config:

feature(
    name='runtime_library_search_directories',
    flag_sets=[
        flag_set(
            actions=[
                'c++-link-executable',
                'c++-link-dynamic-library',
                'c++-link-nodeps-dynamic-library',
            ],
            flag_groups=[
                flag_group(
                    expand_if_available='runtime_library_search_directories',
                    iterate_over='runtime_library_search_directories',
                    flags=['-Wl,-rpath,$ORIGIN/%{runtime_library_search_directories}'],
                )
            ]
        )
    ]
),

(it's #3592 (comment) but starlark syntax)

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[github-actions]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please post @bazelbuild/triage in a comment here and we'll take a look. Thanks!