Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cuGraph-PyG: The "shuffle" Option Does Not Work Correctly #19

Open
Tracked by #4350
alexbarghi-nv opened this issue Jul 26, 2024 · 0 comments
Open
Tracked by #4350

cuGraph-PyG: The "shuffle" Option Does Not Work Correctly #19

alexbarghi-nv opened this issue Jul 26, 2024 · 0 comments
Assignees
Labels
bug Something isn't working
Milestone

Comments

@alexbarghi-nv
Copy link
Member

alexbarghi-nv commented Jul 26, 2024

Currently, shuffle does not work properly, which is good enough for a lot of use cases, but can probably result in lower accuracy when there are many training epochs. This should be resolved by properly permuting the dataset across ranks.

In addition, there needs to be a check that raises a warning if the number of batches on a rank is 0, since this is almost certainly going to cause training to hang (which is normal PyTorch behavior, but the user should be warned first).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant