documentation: need to update how-to-build section of Open MPI for the v5.0.0 release

[hppritcha]

We should make sure this table - https://docs.open-mpi.org/en/v5.0.x/installing-open-mpi/required-support-libraries.html?highlight=prrte

before releasing 5.0.0.

We should also add a section about how to support native launch (esp. via srun) with the 5.0.x release stream.

[qkoziol]

@hppritcha @awlauria - Can you progress this?

[hppritcha]

I will update - https://docs.open-mpi.org/en/v5.0.x/launching-apps/slurm.html to be correct. its inaccurate for the 5.0.x and newer releases.

[qkoziol]

I've tried building the 5.0.x branch with:

hwloc: 1.11.0
Libevent: 2.0.21
PMIx: 4.1.2
PRRTE: 2.0.2

as the table in the docs indicates, and the build fails with:

Making all in .
  CC       errhandler/errhandler.lo
  CC       instance/instance.lo
errhandler/errhandler.c:472:10: error: use of undeclared identifier 'PMIX_ERR_PROC_ABORTED_BY_SIG'
    case PMIX_ERR_PROC_ABORTED_BY_SIG:
         ^
1 error generated.
make[2]: *** [errhandler/errhandler.lo] Error 1
make[2]: *** Waiting for unfinished jobs....
instance/instance.c:443:9: error: use of undeclared identifier 'PMIX_ERR_PROC_ABORTED_BY_SIG'
        PMIX_ERR_PROC_ABORTED_BY_SIG,
        ^
1 error generated.
make[2]: *** [instance/instance.lo] Error 1

That error code is from the PMIx 4.2 branch, so I'm going to try building with the PMIx 4.2.4 release candidate next.

INPUT: Is there any reason to try earlier releases from the openpmix 4.2 branch? I would be inclined to tell people to use the latest release from that branch.

[rhc54]

Configure should have error'd out as OMPI v5 requires a minimum of PMIx v4.2.4 and PRRTE v3.0.1 (when released).

[qkoziol]

Configure should have error'd out as OMPI v5 requires a minimum of PMIx v4.2.4 and PRRTE v3.0.1 (when released).

Agree. Is there any reason to allow earlier versions of PRRTE 4.2.x?

[qkoziol]

FYI - I've verified that the 5.0.x branch currently works with PMIx 4.2.4 and PRRTE 2.0.2. I'll check again when we have frozen the branch for code changes before the 5.0 release.

[rhc54]

Errr...there is no way it should work with PRRTE 2.0.2 - nor would you ever want to advise someone to use that combination. You really need PRRTE 3.0.1 as your minimum supported version.

[qkoziol]

@jsquyres @gpaulsen @awlauria @bwbarrett - Is PRRTE 3.0.1 the best minimum for the OMPI 5.0 release?

[bwbarrett]

If Ralph says that PRRTE 3.0.1 is the right minimum to support, that's what we should do. There's not a big install base of PRRTE outside of Open MPI today, so I don't think there's a huge impact to changing the minimum supported PRRTE version. It looks like we need a CR to change the value in the VERSION file.

We've tried (failed, but tried) to be more conservative bumping up the minimum supported PMIx version, since things like SLURM already are being built with PMIx and systems will have PMIx libraries already installed. I see you tested with PMIx 4.2.4 but our minimum requirement is 4.1.2, and it would be good to understand why we'd have to update.

[rhc54]

PRRTE 3.0.1 is required to support OMPI's v5.0 feature list and your desired cmd line options.

PMIx is a little more complex. You'll need to embed v4.2.4 in order to build the embedded PRRTE, so that is pretty much required. If you are not building PRRTE, then you could possibly get by with some lesser version - you'd have to check and see, depends on the definitions in the OMPI code and how you have protected them.

If you are working with external PRRTE and PMIx, then you could connect OMPI to a lesser PMIx since PMIx is cross-version compatible (and PRRTE would be pointing at PMIx v4.2.4 or above).

Not sure how you would modify configure to handle all those cases, but I imagine it could be done? 🤷‍♂️

[qkoziol]

We've tried (failed, but tried) to be more conservative bumping up the minimum supported PMIx version, since things like SLURM already are being built with PMIx and systems will have PMIx libraries already installed. I see you tested with PMIx 4.2.4 but our minimum requirement is 4.1.2, and it would be good to understand why we'd have to update.

I posted the errors from building w/PMIx 4.1.2 earlier in this thread^^

[qkoziol]

Once PRRTE 3.0.1 is out the door, I'll start updating the docs, submodules, and autoconf things.

[bwbarrett]

We need to have a discussion about whether we should update the PMIx required version or not. I'm getting really nervous about our continued bumping of required PMIx version. When we started 5.0.0x, the goal was the 3.x series (same as OMPI 4.x) and we've long since lost that.

[rhc54]

the goal was the 3.x series (same as OMPI 4.x)

I don't recall that discussion - it is simply impossible given the OMPI v5 feature list. You need something in the v4.x series at the minimum. If you want to cover Sessions and ULFM, then you are talking v4.2.

[qkoziol]

At the 7/11 developer meeting we discussed this issue and thought that having a telecon to discuss how to proceed might be the most productive way to reach agreement. Can anyone who is interested in attending to discuss please look at my post on the Open MPI slack #general channel to schedule a time?

FYI - Slack link https://open-mpi.slack.com/archives/CDGMNGZDY/p1689719829317329
- Poll for meeting time: https://www.when2meet.com/?20635171-REEzV

PS. - closing the meeting time poll @ 5pm CT on 7/19

@rhc54 @bwbarrett @hppritcha @jsquyres @awlauria @gpaulsen @janjust @wenduwan @lrbison

[qkoziol]

I will make some test builds to see which pieces of OMPI require newer PRRTE.

[rhc54]

I will make some test builds to see which pieces of OMPI require newer PRRTE.

I suspect you mean PMIx - OMPI doesn't build against PRRTE and therefore is insensitive (at the build level) to the PRRTE version. OMPI has, however, a functional dependency on PRRTE version for the Sessions and ULFM features.

[rhc54]

Since this issue appears to specifically target the PRRTE support, I doubt there is anything further I can contribute in terms of a meeting. If you want the Sessions and ULFM features to work in OMPI v5, then you need PRRTE v3.0.1. It truly is that simple.

Note that those features will not be available when using a direct launch environment.

[rhc54]

Oh, and do keep in mind that the cmd line options, default placement behaviors, --help output and options, etc are all rather different between PRRTE v2 and v3 series. Going back and forth between them will be difficult.

[qkoziol]

I will make some test builds to see which pieces of OMPI require newer PRRTE.

I suspect you mean PMIx - OMPI doesn't build against PRRTE and therefore is insensitive (at the build level) to the PRRTE version. OMPI has, however, a functional dependency on PRRTE version for the Sessions and ULFM features.

Yes, PMIx, apologies for mixing them

[rhc54]

After reviewing the above plus some notes, and pondering a bit, I believe there is confusion over what is being decided here. Let's try to clear things up a bit.

There are TWO modes for OMPI v5 build and operation:

Mode 1: Direct-launch only
In this mode, OMPI is configured --without-prrte. Thus, mpirun is not built and OMPI is limited to being launched solely by direct launchers such as srun. Since PRRTE is not being built, you have considerable more flexibility on the choice of PMIx version. The MPI code currently cannot support the PMIx v3 series because it lacks protections around definitions included in more recent versions - these were added to support the Sessions and ULFM features.

However, if you are willing to disable those features, then you can extend backward to PMIx v3. I have created a somewhat hacky patch that gets you there (see https://gist.github.com/rhc54/9c615b86f2d43db1c15911390667ab10). You probably would want to clean this up a bunch by adding configure-level logic that disables Sessions and ULFM if the PMIx version is less than 4.0 instead of all the #if clauses I added. If you do, then the changes are fairly small.

Important note: I don't believe that the OMPI you get this way will support debuggers such as DDT or TotalView as it lacks MPIR integration. I suppose you could add that back in if you want - up to you.

Mode 2: OMPI with internal PRRTE
As previously stated, you must use PRRTE v3.0.1 or above to support Sessions and ULFM. Assuming that is a requirement from the community, then you need to embed PMIx v4.2.4 or above in the OMPI v5 distribution as PRRTE v3.0.1 requires that as the minimum PMIx version.

If you want to allow users to configure with an external PMIx and the internal PRRTE, then the PRRTE configure will error out if the external PMIx version is less than v4.2.4.

How you want to capture all that in your table is up to you!

[qkoziol]

At the 7/11 developer meeting we discussed this issue and thought that having a telecon to discuss how to proceed might be the most productive way to reach agreement. Can anyone who is interested in attending to discuss please look at my post on the Open MPI slack #general channel to schedule a time?

FYI - Slack link https://open-mpi.slack.com/archives/CDGMNGZDY/p1689719829317329 - Poll for meeting time: https://www.when2meet.com/?20635171-REEzV

PS. - closing the meeting time poll @ 5pm CT on 7/19

@rhc54 @bwbarrett @hppritcha @jsquyres @awlauria @gpaulsen @janjust @wenduwan @lrbison

Next Wednesday (7/26) at 10-11am CT is the best time, I’ll get an invite out to those who responded. For others, here’s the dial in info:

You have been invited to an online meeting, powered by Amazon Chime.

Click to join the meeting: https://chime.aws/6617440541
Meeting ID: 6617 44 0541
A headset is recommended or you may use your computer’s microphone and speakers.

Call in using your phone:
United States Toll-Free (1): +1 855-552-4463
Meeting ID: 6617 44 0541
One-click Mobile Dial-in (United States (1)): +1 206-462-5569,,,6617440541#
United States (1): +1 206-462-5569
International: https://chime.aws/dialinnumbers/
Dial-in attendees must enter *7 to mute or unmute themselves.

To connect from an in-room video system, use one of the following Amazon Chime bridges:
SIP video system: [email protected] or meet.chime.in
H.323 system: 13.248.147.139 or 76.223.18.152
If prompted enter the Meeting PIN: 6617440541#

Download Amazon Chime at https://aws.amazon.com/chime/download
For information about creating an Amazon Chime account, see https://aws.amazon.com/chime/getting-started

[qkoziol]

More data for this issue: if I disable fault tolerance (with --with-ft=no), I can successfully build and run test programs (with mpirun) on my laptop that use Sessions calls with this set of dependencies:

• hwloc 1.11.0
• libevent 2.0.21
• PMIx 4.1.2
• PRRTE 2.0.2

Here's my configure line: ./configure --disable-mpi-fortran --disable-oshmem-fortran --enable-sphinx --with-libevent=/Users/qkoziol/dev/subspace/OpenMPI/ompi_build_dependencies/install --with-hwloc=/Users/qkoziol/dev/subspace/OpenMPI/ompi_build_dependencies/install --with-pmix=/Users/qkoziol/dev/subspace/OpenMPI/ompi_build_dependencies/install --with-prrte=/Users/qkoziol/dev/subspace/OpenMPI/ompi_build_dependencies/install --prefix=/Users/qkoziol/dev/subspace/OpenMPI/ompi_build_dependencies/install --with-ft=no

(disabling FORTRAN probably doesn't have anything to do with this issue :-) )

I'll give it a try on an actual cluster now, and add more info when I have it.

Update: pasting my simple test program that uses Sessions calls:

#include <assert.h>
#include "mpi.h"
#include <stdio.h>
#include <unistd.h>

void test_allreduce(MPI_Comm comm)
{
	struct {
		double val;
		int   rank;
	} elapsed, min_elapsed, max_elapsed;
	double before, after;
	int mpi_size;
	int mpi_rank;
	int reduce_out;
	int ret;

	/* Use communicator */
	ret = MPI_Comm_size(comm, &mpi_size);
	assert(MPI_SUCCESS == ret);
	ret = MPI_Comm_rank(comm, &mpi_rank);
	assert(MPI_SUCCESS == ret);

	// printf("comm_size = %d, rank = %d\n", mpi_size, mpi_rank);

if(0 == mpi_rank) {
    printf("qak: before allreduce\n");
//    sleep(20);
}
	/* Perform allreduce */
	reduce_out = -1;
	before = MPI_Wtime();
	ret = MPI_Allreduce(&mpi_rank, &reduce_out, 1, MPI_INT, MPI_SUM, comm);
	after = MPI_Wtime();
	assert(MPI_SUCCESS == ret);
if(0 == mpi_rank)
    printf("after allreduce\n");

	/* Set up local elapsed info */
	elapsed.val = after - before;
	elapsed.rank = mpi_rank;

	/* Determine min & max elapsed time (w/rank) from allreduce */
	min_elapsed.val = -1.0;
	min_elapsed.rank = -1;
	ret = MPI_Reduce(&elapsed, &min_elapsed, 1, MPI_DOUBLE_INT, MPI_MINLOC, 0, comm);
	assert(MPI_SUCCESS == ret);
	max_elapsed.val = 1000000.0;
	max_elapsed.rank = -1;
	ret = MPI_Reduce(&elapsed, &max_elapsed, 1, MPI_DOUBLE_INT, MPI_MAXLOC, 0, comm);
	assert(MPI_SUCCESS == ret);

	/* Results of allreduce */
	if(0 == mpi_rank) {
		printf("comm_size = %d\n", mpi_size);
		printf("allreduce out = %d\n", reduce_out);
		printf("rank 0 elapsed = %lf\n", elapsed.val);
		printf("min_elapsed = (%lf, %d), max_elapsed = (%lf, %d)\n", min_elapsed.val, min_elapsed.rank, max_elapsed.val, max_elapsed.rank);
	}


}

int main(int argc, const char *argv[])
{
	MPI_Info sess_info = MPI_INFO_NULL;
	const char mt_key[] = "thread_level";
	const char mt_value[] = "MPI_THREAD_MULTIPLE";
	MPI_Session session = MPI_SESSION_NULL;
	const char pset_name[] = "mpi://WORLD";
	MPI_Group group = MPI_GROUP_NULL;
	MPI_Comm comm = MPI_COMM_NULL;
	int ret;

	ret = MPI_Info_create(&sess_info);
	assert(MPI_SUCCESS == ret);
	ret = MPI_Info_set(sess_info, mt_key, mt_value);
	assert(MPI_SUCCESS == ret);
	ret = MPI_Session_init(sess_info, MPI_ERRORS_RETURN, &session);
	assert(MPI_SUCCESS == ret);

	ret = MPI_Info_free(&sess_info);
	assert(MPI_SUCCESS == ret);

	ret = MPI_Group_from_session_pset(session, pset_name, &group);
	assert(MPI_SUCCESS == ret);

	ret = MPI_Comm_create_from_group(group, "comm_world", MPI_INFO_NULL, MPI_ERRORS_RETURN, &comm);
	assert(MPI_SUCCESS == ret);

	ret = MPI_Group_free(&group);
	assert(MPI_SUCCESS == ret);

	/* Test allreduce with communicator */
	test_allreduce(comm);

	ret =  MPI_Comm_free(&comm);
	assert(MPI_SUCCESS == ret);

	ret = MPI_Session_finalize(&session);
	assert(MPI_SUCCESS == ret);

	return 0;
}

[rhc54]

I'll only reiterate - do not use PRRTE v2.0.2. It is not supported, it has known issues, and you will come to regret it.

You can do it if you like - just don't come to me with any issues you encounter.

[qkoziol]

I'll only reiterate - do not use PRRTE v2.0.2. It is not supported, it has known issues, and you will come to regret it.

You can do it if you like - just don't come to me with any issues you encounter.

Understood

[qkoziol]

I'll only reiterate - do not use PRRTE v2.0.2. It is not supported, it has known issues, and you will come to regret it.

You can do it if you like - just don't come to me with any issues you encounter.

Further testing bears out @rhc54's recommendation here - PRRTE v.2.0.2 does not work for me on AWS clusters.

Therefore: we should be using PRRTE 3.0.x as the minimum for OMPI 5.0. Which means a hard requirement of PMIx 4.2.x, since PRRTE 3.0.x requires at least that version.

[qkoziol]

I'm not certain there's much point in meeting to discuss the options, with this current situation. OMPI 5.0.x needs PRRTE 3.0.x and therefore must also require PMIx 4.2.x.

[rhc54]

OMPI 5.0.x needs PRRTE 3.0.x and therefore must also require PMIx 4.2.x

As I noted earlier, this isn't technically a correct statement. It reflects what should be embedded in OMPI v5, but that isn't the same as what needs to be required during configure. The problem is that we never had a split in that logic for prior releases, but now we do.

How that gets reflected in the configure script and/or the documentation is something you folks will need to decide. Maybe the logic is too complex and you just put a blanket minimum requirement on OMPI v5. Or maybe you add configure logic such that the minimum PMIx requirement changes based on other configure options. Or maybe you come up with some other scheme.

Regardless, it isn't as simple as just defining what should be embedded.

[qkoziol]

Results from our discussion today:

• Update the internal submodules to the latest releases of PRRTE and PMIx, before the OMPI 5.0 release. Must be at least PRRTE 3.0.1 and PMIx 4.2.4, so that Sessions and ULFM are supported for applications launched with internal PRRTE.

• Document that OMPI 5.0 installs that are built "without-prrte", e.g. a "direct run" scenario may not have access to some MPI features. i.e. direct running applications where the launcher (e.g. slurm) is using an older version of PMIx may not support Sessions, ULFM, and debugging tools like DDT & TotalView. This includes directions on how to determine if their direct launch environment has support for the MPI features that the application (or middleware in the application) use, probably with, e.g. "srun pquery". ( <- Verify and document this behavior before release)

[qkoziol]

Note that the description of this behavior is part of #11734

[rhc54]

where the launcher (e.g. slurm) is using an older version of PMIx

Not quite accurate. It has nothing to do with the version of PMIx. The limitation stems from Slurm itself not having implemented the host-level support for Sessions and FT. Ditto for Cray environment.

debugging tools like DDT & TotalView

Again, this has nothing to do with the version of PMIx used by Slurm. The problem is that OMPI v5 only supports PMIx-based attachment methods. So even if srun still supports MPIR (which is the case today - neither they nor Cray support PMIx-based debuggers), the OMPI application process won't stop and wait for an MPIR-based debugger to attach - it will just blow right on by and keep running.

Bottom line: if you want to use a debugger on OMPI v5 applications, you need to get a PMIx-enabled debugger and use mpirun (at least, today - Slurm may someday catch up) to launch the application.

[qkoziol]

The issues around debugging are covered by the https://docs.open-mpi.org/en/v5.0.x/app-debug/parallel-debug.html and https://docs.open-mpi.org/en/v5.0.x/app-debug/mpir-tools.html sections in the docs. I'll submit a PR for updating the version #'s in the "required support libraries" table today.

[qkoziol]

PR to bump the minimum VERSION #'s is up: #11875