Segfault in llvm_gcda_start_file in a mixed C++/Rust program

[Minoru]

I have a program where C++ code calls into Rust code. To calculate test coverage, I compile the Rust part with a Nightly, and C++ part with Clang 8. I then run a C++ test suite. This worked fine until recently, but now segfaults. C++ test suite receives SIGSEGV while writing out the coverage info:

#0  0x0000000002d26c99 in llvm_gcda_start_file ()
#1  0x00000000015e159d in __llvm_gcov_writeout ()
#2  0x00000000015fa856 in __llvm_gcov_flush ()
#3  0x0000000002d28075 in __gcov_flush ()
#4  0x0000000001020c30 in TestHelpers::TempDir::~TempDir (this=0x7fffffffcbf0) at test/test-helpers/tempdir.cpp:36
#5  0x00000000004e5168 in ____C_A_T_C_H____T_E_S_T____10 () at test/cliargsparser.cpp:90
#6  0x0000000000caf89b in Catch::TestInvokerAsFunction::invoke (this=0x3cd9e40) at ./3rd-party/catch.hpp:14222
#7  0x0000000000ca06bf in Catch::TestCase::invoke (this=0x3d6a248) at ./3rd-party/catch.hpp:14065
#8  0x0000000000ca0500 in Catch::RunContext::invokeActiveTestCase (this=0x7fffffffdb50) at ./3rd-party/catch.hpp:12924
#9  0x0000000000c9d946 in Catch::RunContext::runCurrentTest (this=0x7fffffffdb50, Python Exception <class 'gdb.error'> There is no member named _M_dataplus.:
redirectedCout=, Python Exception <class 'gdb.error'> There is no member named _M_dataplus.:
redirectedCerr=) at ./3rd-party/catch.hpp:12897
#10 0x0000000000c9bb97 in Catch::RunContext::runTest (this=0x7fffffffdb50, testCase=...) at ./3rd-party/catch.hpp:12658
#11 0x0000000000ca6e90 in Catch::(anonymous namespace)::TestGroup::execute (this=0x7fffffffdb40) at ./3rd-party/catch.hpp:13252
#12 0x0000000000ca5510 in Catch::Session::runInternal (this=0x7fffffffdf40) at ./3rd-party/catch.hpp:13458
#13 0x0000000000ca4e23 in Catch::Session::run (this=0x7fffffffdf40) at ./3rd-party/catch.hpp:13414
#14 0x0000000000d8c906 in Catch::Session::run<char> (this=0x7fffffffdf40, argc=1, argv=0x7fffffffe1d8) at ./3rd-party/catch.hpp:13136
#15 0x0000000000cde405 in main (argc=1, argv=0x7fffffffe1d8) at test/test.cpp:12

I observe this behaviour with Clang 8 and 9. With Clang 10, things are simply bizarre: the test suite reports numerous SIGSEGVs and then declares that all tests passed; I can't explain such behaviour. With Clang 11, the test suite passes without issue.

bisect-rustc narrowed this down to a recent upgrade to LLVM 11:

searched nightlies: from nightly-2020-08-23 to nightly-2020-08-29
regressed nightly: nightly-2020-08-24
searched commits: from 663d2f5 to 5180f3d
regressed commit: 7ce71c3

bisected with cargo-bisect-rustc v0.5.2

Host triple: x86_64-unknown-linux-gnu
Reproduce with:

cargo bisect-rustc --preserve --with-cargo --host x86_64-unknown-linux-gnu --script ./evaluate.sh

The evaluate.sh is:

#!/bin/sh

export CARGO_INCREMENTAL=0
export RUSTFLAGS='-Zprofile -Ccodegen-units=1 -Copt-level=0 -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort'
export RUSTDOCFLAGS='-Cpanic=abort'
export CC='clang-8'
export CXX='clang++-8'

export TMPDIR=/dev/shm

make distclean
make -j9 PROFILE=1 test
cd test
./test

The project in question can be cloned from https://github.com/newsboat/newsboat.git I failed to write a small reproducer: it appears that I do need the Catch2 test framework and a lot of other code to trigger this.

[ctz]

I also see this. I guess it's a result of LLVM version skew in the first instance, that rustc doesn't ship with the LLVM distribution it links against for use by end-users in the second instance, but the wider problem is that there is no tested, supported coverage tooling for Rust. This is a frustrating situation.

[Minoru]

Almost the same issue, with the same project, this time caused by Rust's upgrade to LLVM 12:

searched nightlies: from nightly-2021-03-01 to nightly-2021-03-10
regressed nightly: nightly-2021-03-05
searched commits: from 476acbf to 45b3c28
regressed commit: 4099208

bisected with cargo-bisect-rustc v0.6.0

Host triple: x86_64-unknown-linux-gnu
Reproduce with:

cargo bisect-rustc --start=2021-03-01 --script ./script.sh

script.sh:

#!/bin/sh

export CXX=clang++-11
export CC=clang-11
export PROFILE=1
export CARGO_INCREMENTAL=0
export RUSTFLAGS='-A warnings -Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -Cpanic=abort'
export RUSTDOCFLAGS='-Cpanic=abort'
export RUST_TEST_THREADS=2
export TMPDIR=/dev/shm/ 

make -j9 all test profclean

cd test
timeout 30 ./test '[this]'

This time it segfaults all over the place: in Rust's stdlib, in gcov, or it just hangs waiting for something inside pthreads.

Compiling C++ with Clang 12 works around the issue. Still can't provide a minimal reproducer because the causes for SIGSEGV are all different and seem to be intertwined with the test harness (Catch2). So building my project seems the only way to reproduce this :(

[Enselic]

Triage: Can you still reproduce this?

[Minoru]

My CI still uses Clang 12 and up-to-date Nightly, and I haven't seen that config fail since March 2021. So at least this doesn't happen every time rustc bumps its LLVM version.

Would you like me to reproduce with Clang 8 and current Nightly?

[ctz]

For my repro of this issue:

rustc doesn't ship with the LLVM distribution it links against for use by end-users in the second instance,

I think this was mostly ameliorated by rustup shipping llvm-tools (#85658).

but the wider problem is that there is no tested, supported coverage tooling for Rust.

I now use https://crates.io/crates/cargo-llvm-cov for this, which is a significantly improved experience over driving the llvm tools myself, and insulates me from worrying about LLVM version skew.

[Enselic]

Sorry for being unclear! I meant "can you reproduce with the latest versions of everything?". Sounds as if the answer is "no"? Then we close this issue I believe?

[Minoru]

Well, the original issue didn't happen when everything was at latest versions -- it only occurred when Clang was older than the LLVM used by rustc. But given that modern rustc doesn't seem to have any problems with (slightly old) Clang 12, I think this issue is no longer important and can indeed be closed.