Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more torch.nn modules #36

Merged
merged 8 commits into from
Sep 20, 2023
Merged

Add more torch.nn modules #36

merged 8 commits into from
Sep 20, 2023

Conversation

davoclavo
Copy link
Contributor

@davoclavo davoclavo commented Jun 28, 2023

Implement more modules from https://pytorch.org/docs/stable/nn.html

TODO

  • Add implementation for more modules
  • Add tests
  • Add documentation using the guide™️
  • Figure out what to do with Sequence + Embeddings (see Support TensorModule of distinct input/output types #34)
  • Break out modules into traits to avoid forgetting an export (akin to ops)
  • Evaluate if using given Conversion is a good approach to replace some extension methods

@sbrunk
Copy link
Owner

sbrunk commented Jul 27, 2023

@davoclavo are you planning to work on this in the near future?

The reason I'm asking is that I want to implement a few transformer models and I need the Embedding module for that.

If not, I might pull your implementation out from here into its own commit (keeping you as author of course) if that's OK for you.

@davoclavo
Copy link
Contributor Author

@sbrunk Hi! Thanks for checking. I have implemented some modules already but haven't had time to add documentation and other things. Let me upload what I have and feel free to pick up from there.

@sbrunk
Copy link
Owner

sbrunk commented Jul 27, 2023

Awesome, thanks!

@davoclavo davoclavo marked this pull request as ready for review July 27, 2023 20:22
@davoclavo
Copy link
Contributor Author

@sbrunk Done, I just uploaded the changes I had (ModuleList and LayerNorm), feel free to pick up the branch as is and add your changes.

@sbrunk sbrunk added design API ergonomics and design enhancement New feature or request and removed design API ergonomics and design labels Jul 30, 2023
Move type parameter in all modules from apply to constructor for consistency and compat with Sequential.
See sbrunk#47 for details about the current design.
@sbrunk
Copy link
Owner

sbrunk commented Sep 18, 2023

Hey @davoclavo I've added tests and examples for the modules and a small fix for LayerNorm actually discovered by a test. :)

I think we should get this merged and tackle the remaining TODOs in later PRs. Would that be ok for you?

@sbrunk sbrunk merged commit 6a933b4 into sbrunk:main Sep 20, 2023
3 checks passed
@davoclavo davoclavo deleted the add-more-modules branch September 26, 2023 01:57
@davoclavo
Copy link
Contributor Author

@sbrunk Sorry for the delay, I was traveling and somehow missed the notification. Thanks for the fixes and the merge! The remaining TODOs can definitely be addressed in future PRs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants