Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add activation functions in jax.nn module #12502

Closed
wants to merge 10,000 commits into from
Closed

Add activation functions in jax.nn module #12502

wants to merge 10,000 commits into from

Conversation

karalleyna
Copy link
Contributor

@karalleyna karalleyna commented Mar 15, 2023 •
edited
Loading

JAX Frontend

  • sigmoid(x)
  • softplus(x)
  • soft_sign(x)
  • silu(x)
  • swish(x)
  • log_sigmoid(x)
  • leaky_relu(x[, negative_slope])
  • hard_sigmoid(x)
  • hard_silu(x)
  • hard_swish(x)
  • hard_tanh(x)
  • elu(x[, alpha])
  • celu(x[, alpha])
  • selu(x)
  • glu(x)

fspyridakos and others added 30 commits March 6, 2023 17:56
@fspyridakos
Fixed argument orders of tf special methods, casting with scalars sho…
88d7d00 
…uld work as expected now
@Daniel4078
fix small formatting error in array/experimental/elementwise
65315d5
@Daniel4078
fix small formatting error in functional/backends/jax/experimental/ac…
a800887 
…tivations
@Daniel4078
fix small formatting error in functional/backends/jax/experimental/la…
8d2235e 
…yers
@Daniel4078
fix small formatting error in functional/backends/numpy/experimental/…
2e5c0b1 
…activations
@Daniel4078
fix small formatting error in functional/backends/numpy/experimental/…
c94f57f 
…elementwise
@Daniel4078
fix small formatting error in functional/backends/tensorflow/experime…
845fcbb 
…ntal/activations
@Daniel4078
fix small formatting error in functional/backends/tensorflow/experime…
69b4d4f 
…ntal/elementwise
@BrookeMa
jax.numpy delivers ediff1d on mathematical functions (ivy-llc#11801)
c98b3c2
@RashulChutani
Merge remote-tracking branch 'origin/master'
40863d9 
# Conflicts:
#	.github/workflows/test-ivy-cron-multi-version.yml
#	.github/workflows/test-ivy-cron.yml
#	ivy/array/experimental/elementwise.py
#	ivy/compiler/utils/XVX.so
#	ivy/compiler/utils/XXI.so
#	ivy/functional/backends/jax/experimental/activations.py
#	ivy/functional/backends/jax/experimental/layers.py
#	ivy/functional/backends/numpy/experimental/activations.py
#	ivy/functional/backends/numpy/experimental/elementwise.py
#	ivy/functional/backends/tensorflow/experimental/activations.py
#	ivy/functional/backends/tensorflow/experimental/elementwise.py
#	ivy/functional/backends/torch/__init__.py
#	ivy/functional/frontends/jax/numpy/mathematical_functions.py
#	ivy/functional/frontends/tensorflow/math.py
#	ivy/functional/frontends/tensorflow/nn.py
#	ivy/functional/ivy/gradients.py
#	ivy_tests/test_ivy/test_frontends/test_jax/test_jax_numpy_math.py
#	ivy_tests/test_ivy/test_frontends/test_tensorflow/test_math.py
#	ivy_tests/test_ivy/test_frontends/test_torch/test_miscellaneous_ops.py
#	ivy_tests/test_ivy/test_frontends/test_torch/test_pointwise_ops.py
#	run_tests.py
#	run_tests_CLI/cron_tests_multi_version.py
@RashulChutani
Resolve Issues in Cron Tests Due to Force Push
e8f5927
@RashulChutani
Update MongoDB Schema for Ivy_tests_copy Database Collection [skip ci]
2aa2f6e
@Aarsh2001
Merge remote-tracking branch 'upstream/master'
c02522f
@RashulChutani
Resolve Bug in run_tests.py (Incorporate Versioning in Mongo DB Schem…
31de3a2 
…a) [skip ci]
@Aarsh2001
Merge remote-tracking branch 'upstream/master'
896c0d4
@RashulChutani
Resolve Bug in run_tests.py due "." not being compatible with MongoDB…
ece79db 
… (Incorporate Versioning in Mongo DB Schema) [skip ci]
@RashulChutani
Resolve Bug in run_tests.py (Incorporate Versioning in Mongo DB Schem…
2389aa1 
…a) [skip ci]
@vedpatwardhan
Added context manager usage to fix error with creating modules with a…
a0725e6 
… torch backend related to uint16 (ivy-llc#11848)
@RashulChutani
Resolve Bug in run_tests.py (Incorporate Versioning in Mongo DB Schem…
6414066 
…a) [skip ci]
@RashulChutani
Merge remote-tracking branch 'origin/master'
edcd29f
@Husienvora
Roll raw_ops tensorflow frontend (ivy-llc#11844)
e675dcd
@RashulChutani
Resolve Bug in run_tests.py (Incorporate Versioning in Mongo DB Schem…
6b6a623 
…a) [skip ci]
@RashulChutani
Merge remote-tracking branch 'origin/master'
87d5227
@RashulChutani
Resolve Bug with Manual Testing Worker [skip ci]
d2a7806
@RashulChutani
Add support to Dynamically Output the Total Number of Tests with Vers…
12c03a5 
…ioning (Backend and Frontend) [skip ci]
@vedpatwardhan
Updated tolerance to fix failing test for elementwise atan2 (ivy-llc#…
97eb1ac 
…11853)
@vedpatwardhan
Added jax.Array as a valid native array with a jax backend (ivy-llc#1…
3babefc 
…1857)
@vedpatwardhan
Added missing on_device argument to test_gradients (ivy-llc#11864)
6e4fe0d
@vedpatwardhan
Updated tolerance to fix failing meta test for test_maml_step_shared_…
831fe69 
…vars (ivy-llc#11870)
@vedpatwardhan
Revert "Updated tolerance to fix failing meta test for test_maml_step…
817b2f0 
…_shared_vars (ivy-llc#11870)" (ivy-llc#11871)

This reverts commit 831fe69.
@karalleyna karalleyna mentioned this pull request Mar 15, 2023
Open
@ivy-leaves ivy-leaves added Array API Conform to the Array API Standard, created by The Consortium for Python Data API Standards Ivy API Experimental Run CI for testing API experimental/New feature or request JAX Frontend Developing the JAX Frontend, checklist triggered by commenting add_frontend_checklist labels Mar 15, 2023
fspyridakos
fspyridakos suggested changes Mar 16, 2023
View reviewed changes
Copy link
Contributor

@fspyridakos fspyridakos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great PR, just some suggeted changes, please remove duplicates in particular and update the docstrings. I'll take another look once you're done. Thanks!

ivy/functional/frontends/torch/nn/functional/non_linear_activation_functions.py Outdated Show resolved Hide resolved
ivy/functional/ivy/experimental/activations.py Outdated
@handle_exceptions
@handle_array_like_without_promotion
@handle_array_function
def sigmoid(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

already implemented

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

ivy/functional/ivy/experimental/activations.py Outdated
@handle_exceptions
@handle_array_like_without_promotion
@handle_array_function
def softplus(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

already implemented

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

ivy/functional/ivy/experimental/activations.py Outdated
@handle_exceptions
@handle_array_like_without_promotion
@handle_array_function
def leaky_relu(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

already implemented

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

ivy/functional/backends/numpy/experimental/activations.py Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For most of these, you can use the native numpy out argument on the last operation to make it use the native out handling. For example if the last operation is division you can do np.divide(x,y,out=out)

ivy/functional/ivy/experimental/activations.py
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some docstrings are incorrect (remain as copies of others) please update them accrdingly

@AnnaTz
Copy link
Contributor

AnnaTz commented Mar 29, 2023 •
edited
Loading

Hi @karalleyna, could you please re-make this PR? It can't get merged as it is now because of the force-push.
Also, could you reserve the functions in the original Iist to avoid other people working on them at the same time.

@ivy-seed ivy-seed added the Stale label Jun 20, 2023
@ivy-seed
Copy link

This PR has been labelled as stale because it has been inactive for more than 7 days. If you would like to continue working on this PR, then please add another comment or this PR will be closed in 7 days.

@ivy-seed ivy-seed closed this Jun 28, 2023
@ivy-seed
Copy link

This PR has been closed because it has been marked as stale for more than 7 days with no activity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fspyridakos fspyridakos fspyridakos requested changes

@AnnaTz AnnaTz Awaiting requested review from AnnaTz

Assignees

@guillesanbri guillesanbri

Labels
Array API Conform to the Array API Standard, created by The Consortium for Python Data API Standards Ivy API Experimental Run CI for testing API experimental/New feature or request JAX Frontend Developing the JAX Frontend, checklist triggered by commenting add_frontend_checklist Stale
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.