Revamp TorchRec intro tutorial #3064
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/3064
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit f799561 with merge base 97b20b3 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
svekars
requested changes
Sep 27, 2024
| kernels and GPU enabled operations to run | ||
|
|
||
| """ | ||
|
|
There was a problem hiding this comment.
Suggested change
| ############################################### |
|
|
||
| """ | ||
|
|
||
| # Install stable versions for best reliability |
There was a problem hiding this comment.
Suggested change
| # Install stable versions for best reliability | |
| # If you do not have the following components in your environment, please install them by running: | |
| # | |
| # .. code-block:: sh |
Comment on lines
17
to
21
|
|
||
| !pip3 install --pre torch --index-url https://download.pytorch.org/whl/cu121 -U | ||
| !pip3 install fbgemm_gpu --index-url https://download.pytorch.org/whl/cu121 | ||
| !pip3 install torchmetrics==1.0.3 | ||
| !pip3 install torchrec --index-url https://download.pytorch.org/whl/cu121 |
There was a problem hiding this comment.
Suggested change
| !pip3 install --pre torch --index-url https://download.pytorch.org/whl/cu121 -U | |
| !pip3 install fbgemm_gpu --index-url https://download.pytorch.org/whl/cu121 | |
| !pip3 install torchmetrics==1.0.3 | |
| !pip3 install torchrec --index-url https://download.pytorch.org/whl/cu121 | |
| # | |
| # !pip3 install --pre torch --index-url https://download.pytorch.org/whl/cu121 -U | |
| # !pip3 install fbgemm_gpu --index-url https://download.pytorch.org/whl/cu121 | |
| # !pip3 install torchmetrics==1.0.3 | |
| # !pip3 install torchrec --index-url https://download.pytorch.org/whl/cu121 |
| @@ -0,0 +1,1081 @@ | |||
| """ | |||
| **Open Source Installation** (For Reference) | |||
There was a problem hiding this comment.
Suggested change
| **Open Source Installation** (For Reference) | |
| Introduction to TorchRec |
| @@ -0,0 +1,1081 @@ | |||
| """ | |||
| **Open Source Installation** (For Reference) | |||
| -------------------------------------------- | |||
There was a problem hiding this comment.
Suggested change
| -------------------------------------------- | |
| ================================== |
|
|
||
| # Install stable versions for best reliability | ||
|
|
||
| !pip3 install --pre torch --index-url https://download.pytorch.org/whl/cu121 -U |
There was a problem hiding this comment.
Suggested change
| !pip3 install --pre torch --index-url https://download.pytorch.org/whl/cu121 -U | |
| # !pip3 install --pre torch --index-url https://download.pytorch.org/whl/cu121 -U |
| # Install stable versions for best reliability | ||
|
|
||
| !pip3 install --pre torch --index-url https://download.pytorch.org/whl/cu121 -U | ||
| !pip3 install fbgemm_gpu --index-url https://download.pytorch.org/whl/cu121 |
There was a problem hiding this comment.
Suggested change
| !pip3 install fbgemm_gpu --index-url https://download.pytorch.org/whl/cu121 | |
| # !pip3 install fbgemm_gpu --index-url https://download.pytorch.org/whl/cu121 |
|
|
||
| !pip3 install --pre torch --index-url https://download.pytorch.org/whl/cu121 -U | ||
| !pip3 install fbgemm_gpu --index-url https://download.pytorch.org/whl/cu121 | ||
| !pip3 install torchmetrics==1.0.3 |
There was a problem hiding this comment.
Suggested change
| !pip3 install torchmetrics==1.0.3 | |
| # !pip3 install torchmetrics==1.0.3 |
| !pip3 install --pre torch --index-url https://download.pytorch.org/whl/cu121 -U | ||
| !pip3 install fbgemm_gpu --index-url https://download.pytorch.org/whl/cu121 | ||
| !pip3 install torchmetrics==1.0.3 | ||
| !pip3 install torchrec --index-url https://download.pytorch.org/whl/cu121 |
There was a problem hiding this comment.
Suggested change
| !pip3 install torchrec --index-url https://download.pytorch.org/whl/cu121 | |
| # !pip3 install torchrec --index-url https://download.pytorch.org/whl/cu121 |
Comment on lines
25
to
27
| # Intro to TorchRec | ||
| # ================= | ||
| # |
There was a problem hiding this comment.
Suggested change
| # Intro to TorchRec | |
| # ================= | |
| # |
svekars
reviewed
Sep 27, 2024
svekars
reviewed
Sep 27, 2024
svekars
reviewed
Sep 30, 2024
.ci/docker/requirements.txt
Outdated
| @@ -70,3 +70,5 @@ pycocotools | |||
| semilearn==0.3.2 | |||
| torchao==0.0.3 | |||
| segment_anything==1.0 | |||
| torchrec | |||
There was a problem hiding this comment.
Can we pin dependencies to specific versions?
There was a problem hiding this comment.
We can do torchrec==0.8 for now, and change to 1.0 once it's out?
svekars
reviewed
Oct 1, 2024
| # Set up environment variables for distributed training | ||
| # RANK is which GPU we are on, default 0 | ||
| os.environ["RANK"] = "0" | ||
| # How many devices in our "world", since Bento can only handle 1 process, 1 GPU |
There was a problem hiding this comment.
We should remove Bento, you can say Colab.
Comment on lines
15
to
17
| * Explore advanced techniques for distributing large embedding tables across multiple GPUs | ||
| .. grid-item-card:: :octicon:`list-unordered;1em;` Prerequisites | ||
| :class-card: card-prerequisites |
There was a problem hiding this comment.
Suggested change
| * Explore advanced techniques for distributing large embedding tables across multiple GPUs | |
| .. grid-item-card:: :octicon:`list-unordered;1em;` Prerequisites | |
| :class-card: card-prerequisites | |
| * Explore advanced techniques for distributing large embedding tables across multiple GPUs | |
| .. grid-item-card:: :octicon:`list-unordered;1em;` Prerequisites | |
| :class-card: card-prerequisites | |
| # following dependencies: | ||
| # | ||
| # .. code-block:: sh | ||
|
|
|
|
||
|
|
||
| ###################################################################### | ||
| # Congrats! |
There was a problem hiding this comment.
Suggested change
| # Congrats! | |
| # Conclusion |
| # Congrats! | ||
| # --------- | ||
| # | ||
| # You have now gone from training a distributed RecSys model all the way |
There was a problem hiding this comment.
Suggested change
| # You have now gone from training a distributed RecSys model all the way | |
| # In this tutorial, you have gone from training a distributed RecSys model all the way |
| # | ||
| # You have now gone from training a distributed RecSys model all the way | ||
| # to making it inference ready. | ||
| # https://github.com/pytorch/torchrec/tree/main/torchrec/inference has a |
There was a problem hiding this comment.
Suggested change
| # https://github.com/pytorch/torchrec/tree/main/torchrec/inference has a | |
| # The `TorchRec repo <https://github.com/pytorch/torchrec/tree/main/torchrec/inference>`__ has a |
| # | ||
| # For more information, please see our | ||
| # `dlrm <https://github.com/facebookresearch/dlrm/tree/main/torchrec_dlrm/>`__ | ||
| # example, which includes multinode training on the criteo terabyte |
There was a problem hiding this comment.
Suggested change
| # example, which includes multinode training on the criteo terabyte | |
| # example, which includes multinode training on the Criteo 1TB |
| # For more information, please see our | ||
| # `dlrm <https://github.com/facebookresearch/dlrm/tree/main/torchrec_dlrm/>`__ | ||
| # example, which includes multinode training on the criteo terabyte | ||
| # dataset, using Meta’s `DLRM <https://arxiv.org/abs/1906.00091>`__. |
There was a problem hiding this comment.
Suggested change
| # dataset, using Meta’s `DLRM <https://arxiv.org/abs/1906.00091>`__. | |
| # dataset using the methods described in `Deep Learning Recommendation Model for Personalization and Recommendation Systems (DLRM) <https://arxiv.org/abs/1906.00091>`__. |
svekars
reviewed
Oct 1, 2024
| @@ -619,3 +619,32 @@ warmup | |||
| webp | |||
| wsi | |||
| wsis | |||
| Meta's | |||
There was a problem hiding this comment.
can you run the sort command in vim (:sort), so it's sorted alphabetically.
svekars
reviewed
Oct 1, 2024
| # Embeddings in PyTorch | ||
| # --------------------- | ||
| # | ||
| # `torch.nn.Embedding <https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html>`__: |
There was a problem hiding this comment.
Suggested change
| # `torch.nn.Embedding <https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html>`__: | |
| # :class:`torch.nn.Embedding`: |
svekars
reviewed
Oct 1, 2024
| # Embedding table where forward pass returns the embeddings themselves as | ||
| # is. | ||
| # | ||
| # `torch.nn.EmbeddingBag <https://pytorch.org/docs/stable/generated/torch.nn.EmbeddingBag.html>`__: |
There was a problem hiding this comment.
Suggested change
| # `torch.nn.EmbeddingBag <https://pytorch.org/docs/stable/generated/torch.nn.EmbeddingBag.html>`__: | |
| # :class:`torch.nn.EmbeddingBag`: |
svekars
reviewed
Oct 2, 2024
svekars
reviewed
Oct 2, 2024
svekars
reviewed
Oct 2, 2024
svekars
added
torchrec
|
need to fix the headings on the side, optimizer heading obfuscates rest of tutorial |
iamzainhuda
reviewed
Oct 2, 2024
| # predictions and reduced model size. For example FP32 (4 bytes) in | ||
| # trained model to INT8 (1 byte) for each embedding weight. This is also | ||
| # necessary given the vast scale of embedding tables, as we want to use as | ||
| # few devices as possible for inference to minimize latency. \* **C++ |
There was a problem hiding this comment.
the bulleted lists don't render as such
that's unfortunately the bug of the theme that we can't currently address. |
svekars
reviewed
Oct 2, 2024
svekars
reviewed
Oct 2, 2024
svekars
reviewed
Oct 2, 2024
svekars
reviewed
Oct 2, 2024
svekars
approved these changes
Oct 2, 2024
svekars
approved these changes
Oct 2, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Revamps TorchRec tutorial for stable release
cc @iamzainhuda