Option to use Julia-provided MPI in machine setup

[johnomotani]

Occasionally it can be useful to use the Julia-provided MPI instead of the system MPI, for example to use Julia-provided binaries for external libraries that depend on MPI (e.g. the MUMPS library that provides MPI-parallelised LU). This PR adds an option to choose the Julia-provided MPI in the machine_setup.sh script.

When using the Julia-provided MPI, you have to use the corresponding mpiexec command. MPI.jl provides a function to generate a link to this command - the machine setup script generates this link in the project directory, so you would run an MPI job like, for example,

./mpiexecjl -np 8 julia --project -O3 --check-bounds=no run_moment_kinetics examples/nonlinear-sound-wave/nonlinear-sound-wave_cheb_split3.toml

[mrhardman]

Thanks for suggesting this feature? Perhaps making a comment in the manual setup to link to https://juliaparallel.org/MPI.jl/stable/usage/ and https://juliaparallel.org/MPI.jl/stable/configuration/#configure_jll_binary could also be helpful?

Have you sucessfully used the mpiexecjl command with a -Jmoment_kinetics.so system image? I am trying this now to see if this solves my issue #275. However, on the first attempt with

julia --project -O3 precompile.jl
.julia/bin/mpiexecjl --project=./ -n 2 julia --project -O3 -Jmoment_kinetics.so -e 'include("moment_kinetics/test/runtests.jl")'

no jobs start, seeming like a hang. I wonder if that is because mpiexecjl only supports -O0. Without the -Jmoment_kinetics.so, the tests run and pass as expected with mpiexecjl, showing that this option of using the native julia MPI should work for manual setup.

EDIT: The hang is only seen with the tests, any job using run_moment_kinetics.jl seems to execute as expected.