UCT/MD/IB: Fix indirect key creation for MT registered memory #9424

ivankochin · 2023-10-17T08:44:46Z

What

Support indirect keys creation for memory regions that were created by multiple treads.

Why ?

MT memory registration can happen (or not) implicitly for large buffers, but the current code doesn't support indirect keys creation for such registration (it always assumes that MR was registered as contig buffer by one thread)

How ?

Create uct_ib_mlx5_devx_reg_ksm_data wrapper that separates KSM creation method by MR flags.

pentschev

Thanks @ivankochin , this seems to resolve the issue I was experiencing. I'm +1 on this change.

test/gtest/uct/ib/test_ib_md.cc

contrib/configure-devel

test/gtest/uct/ib/test_ib_xfer.cc

test/gtest/uct/test_p2p_mix.h

brminich · 2023-10-26T07:44:22Z

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c

@@ -556,6 +567,7 @@ static void uct_ib_mlx5_devx_reg_symmetric(uct_ib_mlx5_md_t *md,
    uint32_t symmetric_rkey;
    ucs_status_t status;

+    ucs_assert(!(memh->super.flags & UCT_IB_MEM_MULTITHREADED));


@tvegas1 do we need to update symmetric key creation? To check whether original memh was registered in multiple threads or not

if ksm (symmetric key) in front of ksm (multi-thread) is not supported, i think we should either:

prevent MT when ((flags & UCT_MD_MEM_SYMMETRIC_RKEY) && (md->flags & UCT_IB_MLX5_MD_FLAG_MKEY_BY_NAME_RESERVE) is true

prevent symmetric key when multi-thread flag is set on the memh

started #9447 to skip altogether mt reg for smkey.

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c

brminich · 2023-10-27T09:26:00Z

@tvegas1, @Artemy-Mellanox can you pls review

ivankochin · 2023-10-27T10:57:21Z

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c

-                                                  (uint64_t)memh->address, 0, 0,
-                                                  "exported key", &cross_mr,
-                                                  &exported_lkey);
+    ucs_assert(!(memh->super.flags & UCT_IB_MEM_MULTITHREADED));


BTW, do someone know should we care about that in case of uct_ib_mlx5_devx_reg_exported_key or not? This call also always assumes that MR was create by single thread.

Should we use uct_ib_mlx5_devx_reg_ksm wrapper? Or assert is OK for now?

i'm not sure it is even supported (ksm + gvmi alias). You can leave an assert for now

brminich · 2023-10-27T11:15:08Z

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c

+        ucs_assertv_always(mr->ksm_data->length > iova_offset,
+                           "mr->ksm_data->length=%ld iova_offset=%u",
+                           mr->ksm_data->length, iova_offset);
+        length = mr->ksm_data->length - iova_offset;


We set iova (which is address + iova_offset) to address field during atomic ksm creation. If I pass full length, then iova + length exceed the registered buffer size and test fails during devx_create_obj() call.

@Artemy-Mellanox WDYT is that OK to pass mr->ksm_data->length - iova_offset as length?

seems we were not doing it previously on uct_ib_mlx5_devx_reg_atomic_key() code path

agree, seems it was mr->ksm_data->length previously, @ivankochin is this another bug fix?

Yes, without the subtraction of the offset I get the following error:

create mkey: window size exceed translations_octword_size

why is it needed now after the change?

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c

test/gtest/uct/ib/test_ib_md.cc

tvegas1 · 2023-10-27T14:30:27Z

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c

+        ucs_assertv_always(mr->ksm_data->length > iova_offset,
+                           "mr->ksm_data->length=%ld iova_offset=%u",
+                           mr->ksm_data->length, iova_offset);
+        length = mr->ksm_data->length - iova_offset;


seems we were not doing it previously on uct_ib_mlx5_devx_reg_atomic_key() code path

yosefe · 2023-10-30T17:14:35Z

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c

-                              size_t length, uint64_t iova, uint32_t mkey_index,
-                              const char *reason, struct mlx5dv_devx_obj **mr_p,
-                              uint32_t *mkey)
+uct_ib_mlx5_devx_reg_ksm_mt(uct_ib_mlx5_md_t *md, int atomic, void *address,


IMO uct_ib_mlx5_devx_reg_ksm_data is a better name - since this function accepts ksm_data structure to register. It's not actually doing a multi-threaded registration

yosefe · 2023-10-30T17:15:55Z

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c

-                         size_t length, int list_size, size_t entity_size,
-                         char *in, uint32_t mkey_index, const char *reason,
-                         struct mlx5dv_devx_obj **mr_p, uint32_t *mkey)
+uct_ib_mlx5_devx_create_ksm(uct_ib_mlx5_md_t *md, int atomic, uint64_t address,


IMO it's worth adding 1-2 lines of documentation to each KSM registration function to document what it's doing in high level. The names are very similar to each other..

yosefe · 2023-10-30T17:17:33Z

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c

+        ucs_assertv_always(mr->ksm_data->length > iova_offset,
+                           "mr->ksm_data->length=%ld iova_offset=%u",
+                           mr->ksm_data->length, iova_offset);
+        length = mr->ksm_data->length - iova_offset;


agree, seems it was mr->ksm_data->length previously, @ivankochin is this another bug fix?

yosefe · 2023-10-30T17:18:12Z

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c

+        ucs_debug("KSM registered memory %p..%p offset 0x%x%s on %s rkey 0x%x",
+                  address, UCS_PTR_BYTE_OFFSET(address, length),
+                  iova_offset, atomic ? " atomic" : "",
+                  uct_ib_device_name(&md->super.dev), *mkey);


maybe put the debug message inside uct_ib_mlx5_devx_reg_ksm_mt and uct_ib_mlx5_devx_reg_ksm_contig, to distinguish between them in the log?

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c

brminich · 2023-11-07T07:43:03Z

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c

+    ucs_assert(exported_lkey != 0);
+    ucs_assert(cross_mr != NULL);


why needed? Does it fix some valgrind error?

Before my patch, both exported_lkey and cross_mr were only defined in the beginning of this function. After adding the logging func in the end of uct_ib_mlx5_devx_reg_ksm_data_contig, compilation on Ubuntu 18.04 and SLES 15 started to fail (all other platforms passed) with message that these variables can be uninitialized.

So I decided to initialize them and check that we change them by the right way by these asserts.

Here is the log of failure:

/__w/1/s/contrib/../src/uct/ib/mlx5/dv/ib_mlx5dv_md.c: In function 'uct_ib_mlx5_devx_reg_exported_key': /__w/1/s/contrib/../src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:487:17: error: 'exported_lkey' may be used uninitialized in this function [-Werror=maybe-uninitialized] return mkey >> 8; ~~~~~^~~~ /__w/1/s/contrib/../src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:2071:14: note: 'exported_lkey' was declared here uint32_t exported_lkey; ^~~~~~~~~~~~~ /__w/1/s/contrib/../src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:2109:25: error: 'cross_mr' may be used uninitialized in this function [-Werror=maybe-uninitialized] memh->cross_mr = cross_mr; ~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~ /__w/1/s/contrib/../src/uct/ib/mlx5/dv/ib_mlx5dv_md.c:2070:29: note: 'cross_mr' was declared here struct mlx5dv_devx_obj *cross_mr; ^~~~~~~~

It appeared after adding uct_ib_mlx5_devx_ksm_log function to uct_ib_mlx5_devx_reg_ksm_data_contig which isn't touch any uninitialized memory. Removing log function solves the issue.

Initialization of these variables seems the best way to solve this issue.

I found that the issue can be solved by removing information about mkey from uct_ib_mlx5_devx_ksm_log function. That information is already printed inside uct_ib_mlx5_devx_reg_ksm in case of successfull registration, so, I think it can be removed here.

brminich · 2023-11-07T07:45:01Z

test/gtest/uct/ib/test_ib_md.cc

+    uct_mem_h memh;
+
+    if ((access_mask & UCT_MD_MEM_ACCESS_REMOTE_ATOMIC) && is_bf_arm()) {
+        UCS_TEST_SKIP_R("FIXME: AMO reg key bug on BF device, skipping");


what is the issue?

I copied that message from test_md::test_reg_mem since it's doing the pretty similar thing. I didn't catch this error by myself.

maybe it's not relevant anymore and can be removed? Can you try remove it and see whether CI passes.
@Artemy-Mellanox are you aware about this issue?

brminich · 2023-11-07T07:48:50Z

test/gtest/uct/ib/test_ib_md.cc

+    comp().comp.func   = dereg_cb;
+    comp().comp.count  = 1;
+    comp().comp.status = UCS_OK;
+    comp().self        = this;


do we really need completion in this test?

completion is required if UCT_MD_MEM_DEREG_FLAG_INVALIDATE is set

test/gtest/uct/ib/test_ib_xfer.cc

test/gtest/uct/test_p2p_mix.cc

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c

test/gtest/uct/ib/test_ib_md.cc

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c

ivankochin · 2023-11-27T10:07:47Z

@Artemy-Mellanox pls review

ivankochin · 2023-11-29T04:18:56Z

@yosefe pls review

yosefe

need to make sure it does not increase testing time

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c

test/gtest/uct/ib/test_ib_md.cc

test/gtest/uct/ib/test_ib_xfer.cc

yosefe

remove use_last_byte from test_p2p_mix
otherwise, LGTM

yosefe

also there is a build failure

yosefe · 2023-12-26T10:48:13Z

test/gtest/uct/ib/test_ib_xfer.cc

+UCS_TEST_P(uct_p2p_mix_test_mt, mix1000_last_byte_offset)
+{
+    /* Alloc 2 chunks buffer, but perform all the operation on the last 8 bytes */
+    run(1000, reg_mt_chunk * 2 - 8, 8);


use ( ) around * expression

yosefe · 2023-12-26T10:50:04Z

test/gtest/uct/uct_test.cc

+    m_rkey.rkey    = other.m_rkey.rkey;
+    m_rkey.handle  = other.m_rkey.handle;
+
+    other.fill_empty();


rename fill_empty -> reset

ivankochin · 2024-01-03T06:01:46Z

/azp run

azure-pipelines · 2024-01-03T06:02:29Z

Azure Pipelines successfully started running 4 pipeline(s).

yosefe · 2024-01-04T10:12:31Z

@ivankochin can u pls squash?

ivankochin · 2024-01-07T17:44:50Z

I did the rebase to resolve the merge conflicts. The only change is add IB_ prefix for all the modified env variables in test_ib_xfer.

yosefe · 2024-01-08T08:23:48Z

I did the rebase to resolve the merge conflicts. The only change is add IB_ prefix for all the modified env variables in test_ib_xfer.

Why not a merge commit?

ivankochin · 2024-01-08T08:28:22Z

I did the rebase to resolve the merge conflicts. The only change is add IB_ prefix for all the modified env variables in test_ib_xfer.

Why not a merge commit?

To patch was already squased so I did it to prevent merge commit creation and squashing it since the merge conflict solution was straightforward for test_ib_xfer. But I completely forgot about required changres in test_ib_md, so I created one more commit yesterday evening.

yosefe · 2024-01-08T09:30:40Z

To patch was already squased so I did it to prevent merge commit creation and squashing it since the merge conflict solution was straightforward for test_ib_xfer. But I completely forgot about required changres in test_ib_md, so I created one more commit yesterday evening.

Even in this case, need to use a merge commit, since there is no good way to review the conflict resolution changes

…ultiple threads

pentschev approved these changes Oct 17, 2023

View reviewed changes

brminich reviewed Oct 17, 2023

View reviewed changes

test/gtest/uct/ib/test_ib_md.cc Outdated Show resolved Hide resolved

ivankochin self-assigned this Oct 19, 2023

ivankochin force-pushed the uct/md/ib/support-mt-ksm-reg-for-indirect-keys branch from c3317ce to a409552 Compare October 20, 2023 05:57

ivankochin commented Oct 24, 2023

View reviewed changes

contrib/configure-devel Outdated Show resolved Hide resolved

ivankochin requested a review from brminich October 26, 2023 06:52

brminich reviewed Oct 26, 2023

View reviewed changes

tvegas1 mentioned this pull request Oct 26, 2023

UCT/IB: Skip multi-thread memory registration for symmetric key #9447

Merged

ivankochin requested a review from Artemy-Mellanox October 27, 2023 10:55

ivankochin commented Oct 27, 2023

View reviewed changes

brminich reviewed Oct 27, 2023

View reviewed changes

tvegas1 reviewed Oct 27, 2023

View reviewed changes

yosefe reviewed Oct 30, 2023

View reviewed changes

ivankochin requested review from brminich and yosefe November 7, 2023 05:38

brminich reviewed Nov 7, 2023

View reviewed changes

brminich mentioned this pull request Nov 7, 2023

UCT/IB/MLX5: Refactor KSM functions and extend logging #9472

Merged

ivankochin commented Nov 8, 2023

View reviewed changes

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c Outdated Show resolved Hide resolved

ivankochin commented Nov 9, 2023

View reviewed changes

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c Outdated Show resolved Hide resolved

ivankochin commented Nov 24, 2023

View reviewed changes

test/gtest/uct/ib/test_ib_md.cc Outdated Show resolved Hide resolved

ivankochin requested a review from brminich November 27, 2023 05:23

brminich reviewed Nov 27, 2023

View reviewed changes

src/uct/ib/mlx5/dv/ib_mlx5dv_md.c Outdated Show resolved Hide resolved

brminich previously approved these changes Nov 27, 2023

View reviewed changes

Artemy-Mellanox previously approved these changes Nov 28, 2023

View reviewed changes

yosefe reviewed Dec 13, 2023

View reviewed changes

ivankochin dismissed stale reviews from Artemy-Mellanox and brminich via 133003a December 15, 2023 08:47

yosefe reviewed Dec 25, 2023

View reviewed changes

yosefe reviewed Dec 26, 2023

View reviewed changes

yosefe previously approved these changes Dec 26, 2023

View reviewed changes

ivankochin force-pushed the uct/md/ib/support-mt-ksm-reg-for-indirect-keys branch from 09d9928 to 173511d Compare December 27, 2023 02:57

ivankochin enabled auto-merge December 27, 2023 02:58

ivankochin dismissed yosefe’s stale review via 4bb8d60 December 27, 2023 08:54

yosefe approved these changes Dec 27, 2023

View reviewed changes

ivankochin force-pushed the uct/md/ib/support-mt-ksm-reg-for-indirect-keys branch from 4bb8d60 to 9622c19 Compare December 28, 2023 06:42

yosefe previously approved these changes Dec 28, 2023

View reviewed changes

ivankochin dismissed yosefe’s stale review via 5ac9620 January 4, 2024 06:07

ivankochin disabled auto-merge January 4, 2024 06:08

yosefe previously approved these changes Jan 4, 2024

View reviewed changes

ivankochin force-pushed the uct/md/ib/support-mt-ksm-reg-for-indirect-keys branch from 5ac9620 to 1706b1c Compare January 5, 2024 08:23

ivankochin dismissed yosefe’s stale review via 967fe61 January 7, 2024 17:42

ivankochin force-pushed the uct/md/ib/support-mt-ksm-reg-for-indirect-keys branch from 1706b1c to 967fe61 Compare January 7, 2024 17:42

yosefe approved these changes Jan 8, 2024

View reviewed changes

UCT/MD/IB: Implement indirect key creation for memory registered by m…

6fbfdea

…ultiple threads

ivankochin force-pushed the uct/md/ib/support-mt-ksm-reg-for-indirect-keys branch from 59cf226 to 6fbfdea Compare January 8, 2024 11:47

yosefe enabled auto-merge January 10, 2024 15:22

yosefe merged commit 00ac237 into openucx:master Jan 10, 2024
118 checks passed

This was referenced Mar 18, 2024

UCP/WIREUP: Consider local distance during slow lanes dropping - v1.16.x #9760

Merged

UCT/MD/IB: Fix indirect key creation for MT registered memory - v1.16.x #9761

Closed

pentschev mentioned this pull request Apr 8, 2024

no remote ep address for lane[3]->remote_lane[3] error with UCX 1.16 rapidsai/ucx-py#1037

Open

		ucs_assert(exported_lkey != 0);
		ucs_assert(cross_mr != NULL);

UCT/MD/IB: Fix indirect key creation for MT registered memory #9424

UCT/MD/IB: Fix indirect key creation for MT registered memory #9424

Conversation

ivankochin commented Oct 17, 2023

What

Why ?

How ?

pentschev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brminich commented Oct 27, 2023 • edited Loading

Choose a reason for hiding this comment

brminich Oct 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yosefe Oct 31, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ivankochin commented Nov 27, 2023

ivankochin commented Nov 29, 2023

yosefe left a comment

Choose a reason for hiding this comment

yosefe left a comment

Choose a reason for hiding this comment

yosefe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ivankochin commented Jan 3, 2024

azure-pipelines bot commented Jan 3, 2024

yosefe commented Jan 4, 2024

ivankochin commented Jan 7, 2024

yosefe commented Jan 8, 2024

ivankochin commented Jan 8, 2024 • edited Loading

yosefe commented Jan 8, 2024

brminich commented Oct 27, 2023 •

edited

Loading

brminich Oct 27, 2023 •

edited

Loading

yosefe Oct 31, 2023 •

edited

Loading

ivankochin commented Jan 8, 2024 •

edited

Loading