unable to use pmix master

[roblatham00]

If i configure mpich like so on summit:

${HOME}/src/mpich/configure --with-device=ch4:ucx \
       --disable-fortran --disable-f77 --disable-f90 \
       CFLAGS=-std=gnu11 \
       --enable-g=all \
       --prefix=/ccs/home/robl/soft/mpich-benvolio \
        --with-pm=none --with-pmix=$(spack location -i pmix)

simple 'cpi' example fails to run, giving an assertion way down in the "hex" routine:

Assertion failed in file /ccs/home/robl/src/mpich/src/util/mpir_pmi.c at line 1105: 0

Backtrace:

(gdb) where
#0  hex (c=201 '\311') at /ccs/home/robl/src/mpich/src/util/mpir_pmi.c:1105
#1  0x0000200000540e2c in decode (size=<error reading variable: Cannot access memory at address 0x7fffea74a3b0>, src=0x4c7ecae0 <Address 0x4c7ecae0 out of bounds>, dest=0x200015960200 <Address 0x200015960200 out of bounds>)
    at /ccs/home/robl/src/mpich/src/util/mpir_pmi.c:1122

I'm not using PMIX from spectrum mpi. I thought I'd try pmix@master from spack. I have no idea how divergent those versions are today.

[raffenet]

Is this reproducible on a regular Linux machine? I'm not sure how compatible OpenPMIx is with the Summit resource manager.

[raffenet]

@jjhursey would you expect a PMIx client from OpenPMIx master to be compatible with the PMIx server on Summit?

[jjhursey]

Are they launching mpich with JSM on Summit?

The PMIx server on Summit is instantiated inside the JSM resource manager. So they would have to launch with jsrun. I have not tried to use PMIx master on the client-side to talk with the installed JSM. However, you can find out the PMIx level JSM (and Spectrum MPI) are using by issuing a jsrun --version. Below is the current output from Summit:

shell$ jsrun --version
jsrun (Job Step Manager) 10.03.01.02rtm0 [Jan 21, 2020] built with PMIx 3.1.4

Report bugs to http://www-03.ibm.com/systems/spectrum-computing/support.html

They are currently running PMIx 3.1.4 so I would try to build MPICH against that release to see if it helps.

[raffenet]

Yes, we have been launching MPICH with jsrun successfully by linking with the PMIx client library bundled with Spectrum MPI. I just wanted to check expectations before digging into this failure with jsrun + PMIx master from Spack. Trying 3.1.4 sounds like a good idea.

[hzhou]

I am going to close this issue and link it to the corresponding discussion - #6142, since it is more on how to work with a specific system rather than a clear issue. If we identify clear issues from the discussion, we can re-open or open a new issue.