Update ROOT's llvm to llvm13. #10294

When we compile incremental inputs we store each input in a separate llvm::Module. In order to preserve the semantics, we make all symbols from each module with external linkage, that is, accessible from other modules. This model works well for almost all symbols, except for the clang-generated ones which have the same name. For example clang stores strings in variables `.strN` which may clash accross modules. Luckily, these strings are used within a single module and thus we can avoid the JIT error by keeping them with their non-external linkage.

It just adds to the complexity and confusion of the addSymbol implementation

When removing code, we need to recover Clang's CodeGen and this happens by iterating over the clang::Decls and the llvm::Module for a given Transaction in the TransactionUnloader. The problem is that now OrcV2 takes the ownership of the llvm::Module from the Transaction for itself and then it can notify us when the module was compiled on demand. Unfortuantely, there is no good way to guarantee that the same module will be attached to the same transaction. This approach allows the TransactionUnloader::unloadModule logic to recover CodeGen. In a longer term, when we switch to partial translation units from clang-repl we might be able to recover CodeGen without needing to know details about the generated llvm::Module.

…mplain.

…r us.

See llvm/llvm-project@159a9f7

Patch by Lang Hames.

This patch also inverts the parameter from ExcludeHostSymbols to IncludeHostSymbols to improve readability. As written in one of the FIXMEs this is a temporary solution which allows us to go further with the llvm-13 upgrade. We can rework the getAddress* functionality to reuse more the ORCV2 infrastructure and more importantly to use that infrastructure in the recommended way.

…make This fixes our -Dbuiltin_cling=Off option.

…oString See llvm/llvm-project@61cdaf6

…ssion. See llvm/llvm-prject@36036aa70ec1df7b51b5d30b2dd8090ad2b6e783

See llvm/llvm-project@772e266

See llvm/llvm-project@fc76b4a

See llvm/llvm-project@bbea4d5

…ryRef

This fixes an assertion.

…ptions. This fixes an option double registration assert.

The in-process symbol generator resolves symbols available in the process and all dlopened libraries. However, upon unresolved symbol, we trigger our lazy loading function creator callbacks which load the relevant library. Then we need another in-process symbol generator to catch the newly created symbol so that the JIT is happy. This solution is suboptimal and in the comments is provided a cleaner implementation, however, this implementation loads libraries for weak symbols that the JIT could generate. This needs to be further investigated.

Apparently cmake sets the cache after the configuration is complete and using FORCE updates the cache at the spot including the relevant generator expressions. This fixes the case where we build ROOT with -Dbuiltin_zlib=Off and then LLVM sanity checks the found zlib issuing: ``` Error evaluating generator expression: $<TARGET_FILE:ZLIB> No target "ZLIB" ```

This should fix the clad builds and unpollute the ROOTSYS/lib.

When creating a dictionary we should either copy the header next to the modulemap file (in the build folder) or use full paths. This patch fixes new diagnostics from such kind: tree/dataframe/test/module.modulemap:3:33: error: cannot emit module dataframe_splitcoll_arrayview: size must be explicitly specified for missing header file "TwoFloats.h" module "TwoFloats.h" { header "TwoFloats.h" export * } ^

When building with C++17, a llvm::StringRef can be implicitly converted to std::string_view. This causes an ambiguity when comparing a StringRef with a const char * because the compiler could either * convert the StringRef into a std::string_view and use the operator== from TString.h, taking std::string_view and const char*, or * construct a StringRef from the const char* and use the opertor== defined in StringRef.h. In TCling.cxx, we want to compare two StringRefs and can construct the StringRefs statically beforehand.

--cuda-gpu-arch aliases --offload-arch, so the option parser will canonicalize to the latter.

This works around a bug + missing support in ORC for constructors with non-void signatures (llvm/llvm-project#54797). Also submitted upstream, see https://reviews.llvm.org/D123370

It doesn't make sense and the JIT crashes, see upstream bug report llvm/llvm-project#54813

"moving a local object in a return statement prevents copy elision" as well as an unused variable and function referenced only from an assert.

Otherwise tests/Autoloading/AutoForwarding.C complains that "an attribute list cannot appear here".

See earlier commit for the Transform.C test.

…angling OSX (due to legacy to disambigate between asm and C symbols) implements an extra linker-level mangling which adds another `_` in the name. However, that is only in the on-disk representation and needs to be dropped upon calling dlsym. OrcV2 gives us a handle to the symbol with starts with `__` allowing us to invert the logic of the symbol search where we drop the `__` for dlsym.

It is (again) necessary to call DefineUsedVTables() to enable (at least) the emission of implicitly defined destructors overriding the virtual destructor in a base class, see the added test.

See earlier commit for the cling/test/.../Transform.C.

Set WeakAnyLinkage instead of ExternalLinkage to tell the JIT linker that it should not error on duplicate symbols.

Starting with version 12, Clang seems to implicitly generate the Derived() constructor, calling Base1() and then Base2(). Older versions and GCC complain that the "call of overloaded 'Derived()' is ambiguous". It's not clear to me what is actually the correct behavior according to the C++ standard, so just delete the unwanted constructor for now.

Fixes (at least) roottest-cling-stl-default-VectorSort and roottest-cling-templateMembers-runoperatorEqual.

Make InterpreterGeneratedCodeDiagnosticsMaybeIncorrect static, which is then turned into a weak symbol by KeepLocalGVPass.

Fixes the build of generated dictionaries.

…t policy. This redoes D103040 in a way that AlwaysIncludeTypeForTemplateArgument = false policy is honored for printing template specialization types. This can be seen for example when printing a canonicalized dependent TemplateSpecializationType which has integral arguments. Signed-off-by: Matheus Izvekov <[email protected]> This fixes: before: root.exe -l -b -q -e 'TClass::GetClass("std::array<int,3>")->Print()' OBJ: TClassarray<int,3UL> after: root.exe -l -b -q -e 'TClass::GetClass("std::array<int,3>")->Print()' OBJ: TClassarray<int,3>

Introduce a backend pass that changes the linkage of external symbols starting with "_ZT" (ie. typeinfo names, typeinfos, and vtables) to weak symbols. This avoids duplicate symbol errors with the new JIT linker. FIXME: This is a hack, we should teach the frontend to emit these only once, or mark all duplicates as available_externally (if that improves performance due to optimizations).

See llvm-project/llvm@50542d504dd869fe1241a9cc987b72ead5a56073

… iteration safe."" This reverts commit 6494f1f. LLVM13 has fixed the issue of deserialization while performing lookups. See llvm/llvm-project@0cb7e7c

This was broken during the llvm upgrade, rewriting this condition.

This behavior is consistent with the construction, and passing a nullptr is a technique to invalidate the cache.

Co-authored-by: Bertrand Bellenot <[email protected]> On Windows, llvm and others want to create `lib/` causing errors. Instead, create the directory explicitly as part of the configuration.

IncrementalExecutor has its own, showing what is requesting the symbols, which is more useful than the diagnostic of Orc. OTOH Orc might emit more errors than just missing symbols, and if there are multiple missing symbols, IncrementalExecutor only shows the first. So enable `Verbose` also for IncrementalJIT, showing the original missing symbols as diagnosed by Orc. This fixes roottest_cling_other_checkMissingSymbolExitCode which fails due to the extra diagnostic error line.

This was a diff we carried around without functional changes.

Replace the existing LazyFunctionCreator interface by a function to add DefinitionGenerators to be passed via the chain Interpreter -> IncrementalExecutor -> IncrementalJIT. Implement a DefinitionGenerator in TCling(Callbacks) to define MaterializationUnits for autoloading symbols. This also allows to remove the double DynamicLibrarySearchGenerator now that the created AutoloadLibraryMUs inject the addresses into the JIT after loading the required library.

Not only applies to C++; the comment about using getGlobalPrefix() for this still holds.

Since cling was ported to LLVM 13, it is required that the `__cuda_register_globals` function and the `__cuda_fatbin_wrapper` and `__cuda_gpubin_handle` variables are unique when defining a CUDA kernel. Otherwise, the JIT is lazy and reuses the compiled version of `__cuda_register_globals`, `__cuda_fatbin_wrapper` and `__cuda_gpubin_handle` from the first CUDA kernel definition for all subsequent CUDA kernel definitions, which in practice means that the PTX code from the first kernel is re-registered each time. Increase the default CUDA SM level to 35 because SM 20 is deprecated or removed in the current CUDA SDK versions.

See llvm/llvm-project@4f3e381

For some reason, neither JIT nor compiled symbols can be resolved without this patch, as if the mangling prefix for win32 is simply wrong for win32?

this is a full path, it will not start with "libNew".

Asking for JIT symbols triggers autoloading. JIT symbols will be resolved by the JIT itself.

OrcV2 uses specific DyLibs to resolve symbols from, we need a dedicated resolver for libCling because its symbols cannot be found from the process. Remove now unused "ExposeHiddenSharedLibrarySymbols".

(cherry picked from commit 872d7c14d8c8e2883ed20a0b6c66a4bbafa4237d)

Fixes root-meta-execUnloading which shows ```==768591== Invalid read of size 8 ==768591== at 0x6EFB10E: llvm::ilist_node_base<false>::getNext() const (ilist_node_base.h:29) ==768591== by 0x6FBF119: llvm::ilist_node_impl<llvm::ilist_detail::node_options<llvm::Function, false, false, void> >::getNext() (ilist_node.h:66) ==768591== by 0x6FBED48: llvm::ilist_iterator<llvm::ilist_detail::node_options<llvm::Function, false, false, void>, false, false>::operator++() (ilist_iterator.h:157) ==768591== by 0x6FBEA20: llvm::simple_ilist<llvm::Function>::begin() (simple_ilist.h:117) ==768591== by 0x6FBE40F: llvm::Module::begin() (Module.h:622) ==768591== by 0x6FBE475: llvm::Module::functions() (Module.h:634) ==768591== by 0x6FBE22F: cling::TransactionUnloader::unloadModule(llvm::Module*) (TransactionUnloader.cpp:175) ==768591== by 0x6FBDF83: cling::TransactionUnloader::RevertTransaction(cling::Transaction*) (TransactionUnloader.cpp:126) ==768591== by 0x6F90CD0: cling::Interpreter::unload(cling::Transaction&) (Interpreter.cpp:1553) ==768591== by 0x6F90E25: cling::Interpreter::unload(unsigned int) (Interpreter.cpp:1574) ==768591== by 0x721A9F2: cling::MetaSema::actOnUndoCommand(unsigned int) (MetaSema.cpp:186) ``` ... ``` ==768591== Address 0x144b0ce0 is 32 bytes inside a block of size 800 free'd ==768591== at 0x484399B: operator delete(void*, unsigned long) (vg_replace_malloc.c:935) ==768591== by 0x6FBC7CA: std::default_delete<llvm::Module>::operator()(llvm::Module*) const (unique_ptr.h:95) ==768591== by 0x6FBC82D: std::__uniq_ptr_impl<llvm::Module, std::default_delete<llvm::Module> >::reset(llvm::Module*) (unique_ptr.h:203) ==768591== by 0x6FBC87E: std::unique_ptr<llvm::Module, std::default_delete<llvm::Module> >::reset(llvm::Module*) (unique_ptr.h:501) ==768591== by 0x6FBC04E: std::unique_ptr<llvm::Module, std::default_delete<llvm::Module> >::operator=(decltype(nullptr)) (unique_ptr.h:434) ==768591== by 0x70E60E7: llvm::orc::ThreadSafeModule::~ThreadSafeModule() (ThreadSafeModule.h:116) ==768591== by 0x936EC79: llvm::orc::IRMaterializationUnit::~IRMaterializationUnit() (Layer.h:31) ==768591== by 0x93707F5: llvm::orc::BasicIRLayerMaterializationUnit::~BasicIRLayerMaterializationUnit() (Layer.h:121) ==768591== by 0x9370811: llvm::orc::BasicIRLayerMaterializationUnit::~BasicIRLayerMaterializationUnit() (Layer.h:121) ==768591== by 0x6E91B45: std::default_delete<llvm::orc::MaterializationUnit>::operator()(llvm::orc::MaterializationUnit*) const (unique_ptr.h:95) ==768591== by 0x6E8F9A3: std::unique_ptr<llvm::orc::MaterializationUnit, std::default_delete<llvm::orc::MaterializationUnit> >::~unique_ptr() (unique_ptr.h:396) ==768591== by 0x935D3C3: llvm::orc::JITDylib::UnmaterializedInfo::~UnmaterializedInfo() (Core.h:1062) ==768591== by 0x935D3DE: void std::_Destroy<llvm::orc::JITDylib::UnmaterializedInfo>(llvm::orc::JITDylib::UnmaterializedInfo*) (stl_construct.h:151) ==768591== by 0x935D26B: void std::allocator_traits<std::allocator<void> >::destroy<llvm::orc::JITDylib::UnmaterializedInfo>(std::allocator<void>&, llvm::orc::JITDylib::UnmaterializedInfo*) (alloc_traits.h:648) ==768591== by 0x935CBA0: std::_Sp_counted_ptr_inplace<llvm::orc::JITDylib::UnmaterializedInfo, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:613) ==768591== by 0x6C39FCE: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:346) ==768591== by 0x6C3CB36: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:1071) ==768591== by 0x6E98CED: std::__shared_ptr<llvm::orc::JITDylib::UnmaterializedInfo, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:1524) ==768591== by 0x6E98D09: std::shared_ptr<llvm::orc::JITDylib::UnmaterializedInfo>::~shared_ptr() (shared_ptr.h:175) ==768591== by 0x932D614: llvm::DenseMapBase<llvm::DenseMap<llvm::orc::SymbolStringPtr, std::shared_ptr<llvm::orc::JITDylib::UnmaterializedInfo>, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr>, llvm::detail::DenseMapPair<llvm::orc::SymbolStringPtr, std::shared_ptr<llvm::orc::JITDylib::UnmaterializedInfo> > >, llvm::orc::SymbolStringPtr, std::shared_ptr<llvm::orc::JITDylib::UnmaterializedInfo>, llvm::DenseMapInfo<llvm::orc::SymbolStringPtr>, llvm::detail::DenseMapPair<llvm::orc::SymbolStringPtr, std::shared_ptr<llvm::orc::JITDylib::UnmaterializedInfo> > >::erase(llvm::orc::SymbolStringPtr const&) (DenseMap.h:307) ==768591== by 0x931028D: llvm::orc::JITDylib::removeTracker(llvm::orc::ResourceTracker&) (Core.cpp:1510) ==768591== by 0x9315060: llvm::orc::ExecutionSession::removeResourceTracker(llvm::orc::ResourceTracker&)::{lambda()#1}::operator()() const (Core.cpp:2155) ==768591== by 0x93150D0: decltype(auto) llvm::orc::ExecutionSession::runSessionLocked<llvm::orc::ExecutionSession::removeResourceTracker(llvm::orc::ResourceTracker&)::{lambda()#1}>(llvm::orc::ExecutionSession::removeResourceTracker(llvm::orc::ResourceTracker&)::{lambda()#1}&&) (Core.h:1326) ==768591== by 0x9315272: llvm::orc::ExecutionSession::removeResourceTracker(llvm::orc::ResourceTracker&) (Core.cpp:2152) ==768591== by 0x93060FD: llvm::orc::ResourceTracker::remove() (Core.cpp:53) ==768591== by 0x70F1D58: cling::IncrementalJIT::removeModule(cling::Transaction const&) (IncrementalJIT.cpp:151) ==768591== by 0x6FBE6CF: cling::IncrementalExecutor::unloadModule(cling::Transaction const&) const (IncrementalExecutor.h:180) ==768591== by 0x6FBDE5B: cling::TransactionUnloader::RevertTransaction(cling::Transaction*) (TransactionUnloader.cpp:119) ==768591== by 0x6F90CD0: cling::Interpreter::unload(cling::Transaction&) (Interpreter.cpp:1553) ==768591== by 0x6F90E25: cling::Interpreter::unload(unsigned int) (Interpreter.cpp:1574) ==768591== by 0x721A9F2: cling::MetaSema::actOnUndoCommand(unsigned int) (MetaSema.cpp:186) ```

Original commit message:" [ORC] Fix weak hidden symbols failure on PPC with runtimedyld Fix "JIT session error: Symbols not found: [ DW.ref.__gxx_personality_v0 ] error" which happens when trying to use exceptions on ppc linux. To do this, it expands AutoClaimSymbols option in RTDyldObjectLinkingLayer to also claim weak symbols before they are tried to be resovled. In ppc linux, DW.ref symbols is emitted as weak hidden symbols in the later stage of MC pipeline. This means when using IRLayer (i.e. LLJIT), IRLayer will not claim responsibility for such symbols and RuntimeDyld will skip defining this symbol even though it couldn't resolve corresponding external symbol. Reviewed By: sgraenitz Differential Revision: https://reviews.llvm.org/D129175 " This patch fixes the same issue for ROOT on ppc64le.

Original commit message: " Teach RuntimeDyld to handle COFF weak references and to consider comdat symbols as weak. Patch by Lang Hames and Sunho Kim! " https://reviews.llvm.org/D138264

…harmless.

The relocation needs to allow for long offsets, as for the JIT, __dso_handle might be outside the shared library. Fixes ``` cling JIT session error: In graph cling-module-10-jitted-objectbuffer, section __TEXT,__StaticInit: relocation target "___dso_handle" at address 0x7fe1ee5052e0 is out of range of Delta32 fixup at 0x108c410bd (___cxx_global_var_initcling_module_10_, 0x108c41090 + 0x2d) [runStaticInitializersOnce]: Failed to materialize symbols: { (main, { $.cling-module-10.__inits.0, __ZN12IncidentTypeL2m1E, __ZN6MarkerD2Ev, __ZN6MarkerD1Ev, ___cxx_global_var_initcling_module_10_.1, __GLOBAL__sub_I_cling_module_10, __ZN6MarkerC2EPKc, ___cxx_global_var_initcling_module_10_.3, __ZN12IncidentTypeL2m3E, __ZN6MarkerC1EPKc, __ZN12IncidentTypeL2m2E, ____orc_init_func.cling-module-10, ___cxx_global_var_initcling_module_10_ }) } ``` as seen no RISC-V and macOS, i.e. with the JITLinker.

Previously, one needed to pass linker-mangled names, which exposes details that clients of IncrementalExecutor should not have to deal with. Instead, use the IR name and do the linker-mangling in IncrementalJIT::addOrReplaceDefinition(). This fixes the lack of static destruction on macOS, visible e.g. in the test failure of roottest/cling/staticinit/ROOT-7775.

This helps with debugging why a certain header does not end up in a module that is currently generated.

It is used by core/base and causes laying violations with modules, see module ROOT_Foundation_C which needs to contain ThreadLocalStorage.h.

This is still required for proper exception handling support. See commits 3f74182 and a7b0b3e by Philippe for details on the problem and the original code, which I was able to significantly simplify with the new JIT infrastructure. For the moment, this disables JITLink even on platforms that had it active by default (notably macOS). This will be reintroduced at a later point, which will require a memory manager implementation for the JITLink interface.

This is required until CallFunc is informed about unloading, and can re-generate the wrapper (if the decl is still available).

If the Decl pointers are not identical, declaresSameEntity will check the canonical Decls. This fixes the df021_createTGraph tutorial on CentOS 8. Upstream discussion in https://reviews.llvm.org/D137787

It finds a false positive in llvm::DataLayout::reset accessing DefaultAlignments while loading the Net library. At this point, Cling and LLVM have definitely been initialized. This is a known limitation of the order check, which is why it was disabled by default. Remove our config to enable it for now to allow checking the LLVM upgrade with AddressSanitizer instrumentation.

This is only used to determine the architecture for OpenMP offloading, which we are not interested in.

As https://johannst.github.io/notes/development/symbolver.html points out, "@@" just means "default version". We want to skip any versioned GCC/glibc/ libstdc++ symbol. Significantly reduces appetite for searching symbols. Then also skip wek symbols: even if they are unresolved we will happily not have them if they have not been loaded so far. Typical case is `__gmon_start__`.

When creating orec-Symbols for dylib symbols, reloading the dylib might mean a change in symbol (address). So unloading a dylib means we need to unload the orc-Symbol. This is implemented through resource-tracking the symbols as provided by DynamicLibrarySearchGenerator. Actually, as DynamicLibrarySearchGenerator does not support resource tracking, it is implemented in a near-copy of DynamicLibrarySearchGenerator, RTDynamicLibrarySearchGenerator, which uses the transaction of the most recent module for the ResourceTracker.

The splitting code requires the full string to be null-terminated. Before, random bytes were attached to the last option.

on macOS11, stressInterpreter fails with not re-emitting _ZL6strstrUa9enable_ifILb1EEPKcS0_. It has internal linkage: originally deferred, it needs a chance to be re-emitted. Simplify this to include anything that was deferred and is either weak-for-linker or internal, to hopefully match all cases.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update ROOT's llvm to llvm13. #10294

Update ROOT's llvm to llvm13. #10294

Commits on Dec 9, 2022