openshmem-parcelport initial import #6308

ct-clmsn · 2023-07-25T01:27:14Z

Any background context you want to provide?

Implements an OpenSHMEM parcelport. This PR adds cmake-based (via pkg-config) OpenSHMEM configuration for the sandia-openshmem and the ucx-openshmem.

OpenSHMEM creates a distributed shared segment on each locality. The following flags were added to cmake.

-DHPX_WITH_PARCELPORT_OPENSHMEM=ON
-DHPX_WITH_PARCELPORT_OPENSHMEM_CONDUIT={sos, ucx}

The sos option is for Sandia-OpenSHMEM. The ucx option is for the UCX-OpenSHMEM.

Both Sandia OpenSHMEM and UCX-OpenSHMEM should be compiled with --enable-shared to make sure their OpenSHMEM builds are shared libraries. The GASNet-parcelport established the foundation for this implementation. Each locality gets a page of symmetric memory, equal to what is provided in GASNet, for communication.

Checklist

Not all points below apply to all pull requests.

I have added a new feature and have added tests to go along with it.

StellarBot · 2023-07-25T01:44:00Z

Performance test report

HPX Performance

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR	PARALLEL_EXECUTOR	SCHEDULER_EXECUTOR
For Each	(=)	??	-

Info

Property	Before	After
HPX Datetime	2023-05-10T12:07:53+00:00	2023-07-25T01:32:54+00:00
HPX Commit	`dcb5415`	`a4397d3`
Clustername	rostam	rostam
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Datetime	2023-05-10T14:50:18.616050-05:00	2023-07-24T20:41:08.096903-05:00
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1
Envfile

Comparison

BENCHMARK	NO-EXECUTOR
Future Overhead - Create Thread Hierarchical - Latch	=

Info

Property	Before	After
HPX Datetime	2023-05-10T12:07:53+00:00	2023-07-25T01:32:54+00:00
HPX Commit	`dcb5415`	`a4397d3`
Clustername	rostam	rostam
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Datetime	2023-05-10T14:52:35.047119-05:00	2023-07-24T20:43:22.598094-05:00
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1
Envfile

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR	PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR	SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR
Stream Benchmark - Add	(=)	(=)	(=)
Stream Benchmark - Scale	(=)	(=)	(=)
Stream Benchmark - Triad	(=)	=	(=)
Stream Benchmark - Copy	(=)	(=)	(=)

Info

Property	Before	After
HPX Datetime	2023-05-10T12:07:53+00:00	2023-07-25T01:32:54+00:00
HPX Commit	`dcb5415`	`a4397d3`
Clustername	rostam	rostam
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Datetime	2023-05-10T14:52:52.237641-05:00	2023-07-24T20:43:40.002408-05:00
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1
Envfile

Explanation of Symbols

Symbol	MEANING
=	No performance change (confidence interval within ±1%)
(=)	Probably no performance change (confidence interval within ±2%)
(+)/(-)	Very small performance improvement/degradation (≤1%)
+/-	Small performance improvement/degradation (≤5%)
++/--	Large performance improvement/degradation (≤10%)
+++/---	Very large performance improvement/degradation (>10%)
?	Probably no change, but quite large uncertainty (confidence interval with ±5%)
??	Unclear result, very large uncertainty (±10%)
???	Something unexpected…

StellarBot · 2023-07-25T14:29:26Z

Performance test report

HPX Performance

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR	PARALLEL_EXECUTOR	SCHEDULER_EXECUTOR
For Each	(=)	??	-

Info

Property	Before	After
HPX Datetime	2023-05-10T12:07:53+00:00	2023-07-25T14:17:40+00:00
HPX Commit	`dcb5415`	`8626ce7`
Datetime	2023-05-10T14:50:18.616050-05:00	2023-07-25T09:26:02.549234-05:00
Clustername	rostam	rostam
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Envfile
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1

Comparison

BENCHMARK	NO-EXECUTOR
Future Overhead - Create Thread Hierarchical - Latch	=

Info

Property	Before	After
HPX Datetime	2023-05-10T12:07:53+00:00	2023-07-25T14:17:40+00:00
HPX Commit	`dcb5415`	`8626ce7`
Datetime	2023-05-10T14:52:35.047119-05:00	2023-07-25T09:28:17.273190-05:00
Clustername	rostam	rostam
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Envfile
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1

Comparison

BENCHMARK	FORK_JOIN_EXECUTOR_DEFAULT_FORK_JOIN_POLICY_ALLOCATOR	PARALLEL_EXECUTOR_DEFAULT_PARALLEL_POLICY_ALLOCATOR	SCHEDULER_EXECUTOR_DEFAULT_SCHEDULER_EXECUTOR_ALLOCATOR
Stream Benchmark - Add	=	=	=
Stream Benchmark - Scale	(=)	(=)	(=)
Stream Benchmark - Triad	(=)	(=)	(=)
Stream Benchmark - Copy	(=)	=	(=)

Info

Property	Before	After
HPX Datetime	2023-05-10T12:07:53+00:00	2023-07-25T14:17:40+00:00
HPX Commit	`dcb5415`	`8626ce7`
Datetime	2023-05-10T14:52:52.237641-05:00	2023-07-25T09:28:34.633886-05:00
Clustername	rostam	rostam
Hostname	medusa08.rostam.cct.lsu.edu	medusa08.rostam.cct.lsu.edu
Envfile
Compiler	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1	/opt/apps/llvm/13.0.1/bin/clang++ 13.0.1

Explanation of Symbols

Symbol	MEANING
=	No performance change (confidence interval within ±1%)
(=)	Probably no performance change (confidence interval within ±2%)
(+)/(-)	Very small performance improvement/degradation (≤1%)
+/-	Small performance improvement/degradation (≤5%)
++/--	Large performance improvement/degradation (≤10%)
+++/---	Very large performance improvement/degradation (>10%)
?	Probably no change, but quite large uncertainty (confidence interval with ±5%)
??	Unclear result, very large uncertainty (±10%)
???	Something unexpected…

Signed-off-by: Johan511 <[email protected]>

The major changes include: - Necessary changes to CMake files to accommodate changes due to the split of the original LCI library into two libraries (LCI and LCT). - A new performance counters and logging infrastructure based on LCT (currently only applied to the lci pp). - Controlled by CMake variables HPX_WITH_PARCELPORT_LCI_PCOUNTER and HPX_WITH_PARCELPORT_LCI_LOG. The default is OFF. - New LCI parcelport configurations: - reg_mem: whether to explicitly register memory buffers for long messages (value 1) or just let LCI register them on the fly (value 0). The default is 1. - ndevices: how many LCI devices (low-level network contexts) to use. The default is 2. - ncomps: how many completion managers to use. The default is 1.

ct-clmsn · 2023-10-24T01:59:17Z

@hkaiser is it ok if this branch is closed in favor of a newer branch build around the current version of HPX? the rebase is a little too extensive.

hkaiser · 2023-10-24T11:40:04Z

@hkaiser is it ok if this branch is closed in favor of a newer branch build around the current version of HPX? the rebase is a little too extensive.

Sure, whatever works for you. You could also simply force push your new version to this PR (overwrite what's there), though.

ct-clmsn requested review from aurianer, msimberg, biddisco and hkaiser as code owners July 25, 2023 01:27

hkaiser added type: enhancement category: parcel transport type: compatibility issue labels Jul 27, 2023

hkaiser added this to the 1.10.0 milestone Jul 27, 2023

hkaiser and others added 20 commits October 19, 2023 11:19

Fixing formatting

49f45d7

changed chat link to discord

1c8db2b

Signed-off-by: Johan511 <[email protected]>

added cpp slack channel

7c4ffa2

Signed-off-by: Johan511 <[email protected]>

Make coverage work with Jenkins Github Branch Source plugin

43f216c

Cleanup Jenkins scripts

707ba76

Fix missing links for header files

14d4308

Fix header file links

f717f7a

Enable using std::reference_wrapper with executor parameters

e95d895

Adding --exclusive to launching tests on rostam

78941a0

Moving hpx::threads::run_as_xxx to namespace hpx

23792b8

move is_(trivially_)relocatable to experimental

93b1d3c

type_support::uninitialized_relocate -> primitive

f07d132

hpx seq-par uninialized_relocate(_n) algorithm

7843af1

tests for par-seq uninitialized_relocate(_n)

4448ed5

tests for relocation (nitpicks)

4384a5f

clang-format + namespaces

3605199

fix unrelated failing test

f0d170c

move is_contiguous_iterator to type_support, support array iterators

f9394f8

detect vec, arr, string w/out causing hard errors

73465e2

Fix broken links docs

f4c0ff3

ct-clmsn and others added 26 commits October 19, 2023 12:39

attempt to correct linking issue in builder/ctest

f92a22d

changing builders to test against udp conduit

c6f6025

changing builders to test against udp conduit

96c63aa

added 'maybe unused'

c582982

fixed MPI_CC detection issue

3f5381e

added maybe_unused

630b1a3

added maybe_unused

ee9f773

possible mpi_cc fix

9cecf4d

possible testscript fix

e47935e

cmake format issue resolved

5886475

Fixing inspect problems

5fd114b

Fixing link to dependency report

03074b2

reloc test wrong atomic usage

ddf5957

reloc test, loosen test requirements

47e8a78

More cleaning up for module levels 19-20

668e7de

working issue with MPI_CC / CC conflict in automake

888445b

working issue with MPI_CC / CC conflict in automake

3a23d52

working issue with MPI_CC / CC conflict in automake

a1729a2

initial import

51412cc

initial import of openshmem parcelport

b0eb140

added openshmem builds support

6e14d86

multiple fixes for dependency fetch, compile, install

c80f121

rm'd lingering gasnet references

1ef2004

fixed formatting

9c1adeb

started fixing formatting issues

2f63c16

ct-clmsn closed this Oct 25, 2023

ct-clmsn mentioned this pull request Oct 25, 2023

openshmem parcelport #6376

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

openshmem-parcelport initial import #6308

openshmem-parcelport initial import #6308

ct-clmsn commented Jul 25, 2023 •

edited

Loading

StellarBot commented Jul 25, 2023

HPX Performance

Comparison

Info

Comparison

Info

Comparison

Info

Explanation of Symbols

StellarBot commented Jul 25, 2023

HPX Performance

Comparison

Info

Comparison

Info

Comparison

Info

Explanation of Symbols

ct-clmsn commented Oct 24, 2023

hkaiser commented Oct 24, 2023

openshmem-parcelport initial import #6308

openshmem-parcelport initial import #6308

Conversation

ct-clmsn commented Jul 25, 2023 • edited Loading

Any background context you want to provide?

Checklist

StellarBot commented Jul 25, 2023

HPX Performance

Comparison

Info

Comparison

Info

Comparison

Info

Explanation of Symbols

StellarBot commented Jul 25, 2023

HPX Performance

Comparison

Info

Comparison

Info

Comparison

Info

Explanation of Symbols

ct-clmsn commented Oct 24, 2023

hkaiser commented Oct 24, 2023

ct-clmsn commented Jul 25, 2023 •

edited

Loading