Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor(ivy, torch-frontend): Extends ivy.multi_head_attention and shortens torch_frontend.multi_head_attention_forward #23131

Merged
merged 84 commits into from
Sep 27, 2023
Merged

refactor(ivy, torch-frontend): Extends ivy.multi_head_attention and shortens torch_frontend.multi_head_attention_forward #23131

merged 84 commits into from
Sep 27, 2023

Conversation

AnnaTz
Copy link
Contributor

@AnnaTz AnnaTz commented Sep 6, 2023 •
edited
Loading

Extending ivy.multi_head_attention to the superset, and shortening torch_frontend.multi_head_attention_forward by using the updated ivy.multi_head_attention.

Close #23074

@github-actions
Copy link
Contributor

github-actions bot commented Sep 6, 2023

Thanks for contributing to Ivy! 😊👏
Here are some of the important points from our Contributing Guidelines 📝:
1. Feel free to ignore the run_tests (1), run_tests (2), … jobs, and only look at the display_test_results job. 👀 It contains the following two sections:
- Combined Test Results: This shows the results of all the ivy tests that ran on the PR. ✔️
- New Failures Introduced: This lists the tests that are passing on main, but fail on the PR Fork. Please try to make sure that there are no such tests. 💪
2. The lint / Check formatting / check-formatting tests check for the formatting of your code. 📜 If it fails, please check the exact error message in the logs and fix the same. ⚠️🔧
3. Finally, the test-docstrings / run-docstring-tests check for the changes made in docstrings of the functions. This may be skipped, as well. 📚
Happy coding! 🎉👨‍💻

@AnnaTz AnnaTz changed the title refactor(ivy, frontends): Extends ivy.multi_head_attention and shortens torch_frontend.multi_head_attention_forward refactor(ivy, torch-frontend): Extends ivy.multi_head_attention and shortens torch_frontend.multi_head_attention_forward Sep 6, 2023
@ivy-leaves ivy-leaves added PyTorch Frontend Developing the PyTorch Frontend, checklist triggered by commenting add_frontend_checklist Ivy Functional API labels Sep 6, 2023
@ivy-leaves ivy-leaves added the Array API Conform to the Array API Standard, created by The Consortium for Python Data API Standards label Sep 7, 2023
@AnnaTz
Merge branch 'main' into mha
236377b
@AnnaTz
Merge branch 'main' into mha
116c891
@AnnaTz
corrected pre_embed_dim vs embed_dim
c88852d
@AnnaTz
avoid overflows
b944063
@AnnaTz
fix pre-embedding bug
8794146
@AnnaTz
torch doesn't support self-attention
b37e5bd
@AnnaTz
Merge branch 'main' into mha
65ba5b7
@AnnaTz
avoid overflow
d637691
@AnnaTz
corrected argument order
a1076d0
@AnnaTz
values won't be the same when training due to random dropout
9d38d95
@AnnaTz
updated array ranges in the test
db6a137
@AnnaTz
fixed some bugs of embedding vs pre-embedding dims
cd63554
@AnnaTz
forgot to remove code when moving it
03acf9d
@AnnaTz
wrong dim expanded in torch backend
9192fa0
@AnnaTz
shortened the partial mixed handler check
71a4707
@AnnaTz
native torch always returns a tuple
9548879
@AnnaTz
fixed attention mask bugs
83c3671
@AnnaTz
have same_pre_embed_dim affect all cases in the strategy, so that the…
6cbb35d 
… torch backend is properly tested
AnnaTz and others added 24 commits September 20, 2023 11:46
@AnnaTz
force original mask dtypes
fc31f2e
@AnnaTz
new detailed docstring
4b4e8a2
@AnnaTz
tolerance
859bbdd
@AnnaTz
finally fixed the dimension referencing issue in the static case
a5ce167
@AnnaTz
previous commit continued
1e9daf2
@AnnaTz
fixed bug of wrong dtypes passed to test_frontend_function
08d42a6
@AnnaTz
Merge branch 'main' into mha
acb5abb
@AnnaTz
fixed attention mask, numeric masks are added to the weights
baf0e21
@AnnaTz
emb_dim should be integer
7952491
@AnnaTz
refactored the shaping of the matrix multiplications
2f00897
@AnnaTz
similar attention mask fix for torch backend
bf56f48
@AnnaTz
make scale positive
9308c66
@AnnaTz
mask refactor
a2d2a28
@AnnaTz
key_padding_mask needs to be converted to float like attention_mask
7907596
@AnnaTz
corrected case of is_causal and need_weights in the frontend
c2e594f
@AnnaTz
fixed is_causal + more masking bugs
827086e
@ivy-branch
🤖 Lint code
fda3ee1
@AnnaTz
added argument batch_first
dca30d6
@AnnaTz
Merge branch 'main' into mha
b350187
@ivy-branch
🤖 Lint code
72380d1
@Madjid-CH @AnnaTz
test(backend handler): added tests for inplace update warnings. (#26067)
803e2a4
@AnnaTz
fix(ivy): Added missing wrapper import
8da76fe
@AnnaTz
minor changes
c2118d6
@AnnaTz
refactored & fixed static_k/static_v
47be090
@AnnaTz AnnaTz marked this pull request as ready for review September 27, 2023 15:13
@AnnaTz AnnaTz merged commit dfe5ea0 into ivy-llc:main Sep 27, 2023
91 of 136 checks passed
@AnnaTz AnnaTz deleted the mha branch October 9, 2023 11:10
druvdub pushed a commit to druvdub/ivy that referenced this pull request Oct 14, 2023
@AnnaTz @druvdub
refactor(ivy, torch-frontend): Extends ivy.multi_head_attention and s…
fdab979 
…hortens torch_frontend.multi_head_attention_forward (ivy-llc#23131)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@juliagsy juliagsy

Labels
Array API Conform to the Array API Standard, created by The Consortium for Python Data API Standards Ivy Functional API PyTorch Frontend Developing the PyTorch Frontend, checklist triggered by commenting add_frontend_checklist
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

multi_head_attention_forward - 70 LOC
5 participants
@AnnaTz @juliagsy @ivy-leaves @ivy-branch @Madjid-CH