Kokkos.jl: A Kokkos wrapper for Julia

A Julia wrapper around the popular Kokkos C++ library, using CxxWrap.jl.

This package allows to create Kokkos::View objects, use them as an Array in Julia, and call a C++ Kokkos library through ccall with those views. Most basic functionalities of Kokkos (initialization, views, subviews, copies, fences, memory and execution spaces) are available. With MPI.jl it is possible to use any view with MPI seamlessly.

Kokkos.jl does not currently offer the possibility to code Kokkos kernels in Julia, they must be written in a separate C++ shared library. If the library you want to use is configured with CMake, it is possible to configure the project with Kokkos.jl.

This package relies on a wrapper library which is compiled when initializing Kokkos, which is configured with the CMake and Kokkos options set in the configuration options. Because it is not pre-compiled as an artifact, this maximizes the flexibility of usage of Kokkos.jl. However, most Kokkos functions (and views) are compiled separately on demand, when their respective Julia method is called for the first time. The resulting shared library is then cached for the next session.

Kokkos.jl currently supports Kokkos v3.7, v4.0 and above. All Kokkos backends should be supported by this package, but not all of them were tested (yet).

Supported functionalities

• ✅ Kokkos::initialize, Kokkos::finalize and Kokkos::InitializationSettings
• ✅ Kokkos::View, Kokkos::View<T, MyLayout, SomeMemorySpace> and Kokkos::view_alloc
• ❌ Kokkos::MemoryTraits (planned)
• ✅ Kokkos::create_mirror, Kokkos::create_mirror_view
• ✅ Kokkos::deep_copy
• ✅ Kokkos::subview
• ❌ Kokkos::resize, Kokkos::realloc (planned)
• ✅ Kokkos::fence
• ✅ All execution spaces (Kokkos::OpenMP, Kokkos::Cuda...) and memory spaces (Kokkos::HostSpace, Kokkos::CudaSpace...)
• ❌ All parallel patterns (Kokkos::parallel_for, Kokkos::parallel_reduce, Kokkos::parallel_scan), reducers, execution policies and tasking
• ❌ Atomics
• ❌ All containers (Kokkos::DualView, Kokkos::ScatterView...) (planned)
• ❌ SIMD
• ❌ View hooks

Tested backends

• ✅ Kokkos::Serial
• ✅ Kokkos::OpenMP
• ❌ Kokkos::Threads
• ❌ Kokkos::HPX
• ❌ Kokkos::OpenMPTarget
• ✅ Kokkos::Cuda + interop with CUDA.jl
• ✅ Kokkos::HIP + interop with AMDGPU.jl
• ❌ Kokkos::SYCL
• ❌ Kokkos::OpenACC

Known issues

• The NVCC compiler is unable to compile the wrapper library. Use Clang instead (Clang-11 is the version used to test this backend).
• Kokkos::Cuda ERROR: Failed to call Kokkos::Cuda::finalize() message when exiting Julia: Kokkos.finalize() is not yet called automatically upon quitting Julia, as it would require to call the finalizers of all views before doing so.
• Memory leaks on GPU: this is a side effect of Julia's GC which cannot manage device memory. From Julia's POV, a Kokkos.View is only a pointer in the host memory. Calling GC.gc(true) will fix the issue.