Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve allgather functions #9649

Merged
merged 18 commits into from
Oct 12, 2023
Merged

Improve allgather functions #9649

merged 18 commits into from
Oct 12, 2023

Conversation

rongou
Copy link
Contributor

@rongou rongou commented Oct 10, 2023

  • Change the interface to take vectors to make it easier to call
  • Implemented AllgatherV
  • Implemented allgather of strings, needed for syncing column-split feature names/types

@rongou
Copy link
Contributor Author

rongou commented Oct 11, 2023

@trivialfis

Copy link
Member

@trivialfis trivialfis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good to me.

  • Could you please look into the test failure?
  • Is there a way to consolidate the interface into something more compact? I'm working on revamping the coll implementation and find it quite difficult to work on at the moment. There are many variants of the interface, some supported by GPU some others are not.

@rongou
Copy link
Contributor Author

rongou commented Oct 11, 2023

Not sure about the test failure. I can't reproduce it on my desktop, even after downgrade gcc from 12 to 9. Any ideas?

Yeah the current interfaces evolved from the original rabit/nccl implementations. Need to invest some additional effort to try to unify them.

@trivialfis
Copy link
Member

trivialfis commented Oct 12, 2023

@rongou I think it's a sanitizer error. You can enable it through the cmake flag USE_SANITIZER=ON and ENABLED_SANITIZERS=address;undefined. For other related flags to get CUDA to work and show logging, see doc/contrib/unit_tests.rst.

@rongou
Copy link
Contributor Author

rongou commented Oct 12, 2023

@trivialfis Finally got the CI to pass. Please take another look. Thanks!

@trivialfis trivialfis merged commit e164d51 into dmlc:master Oct 12, 2023
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants