Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Nondeterministic Hypothesis data generation #28733

Open
4 tasks
jacksondm33 opened this issue Apr 11, 2024 · 0 comments
Open
4 tasks

[Bug]: Nondeterministic Hypothesis data generation #28733

jacksondm33 opened this issue Apr 11, 2024 · 0 comments
Labels
Bug Report Report bugs detected in Ivy.

Comments

@jacksondm33
Copy link
Contributor

Bug Explanation

Some test data generated via Ivy's helper strategies is nondeterministic (between runs) when using the derandomize option with Hypothesis. Specifically, this appears to be caused by the conversion of sets of string values to lists/tuples, which results in a nondeterministic ordering. This behavior is ultimately caused by Python's hashing randomization:

By default, the hash() values of str and bytes objects are “salted” with an unpredictable random value. Although they remain constant within an individual Python process, they are not predictable between repeated invocations of Python.

This most notably occurs in the get_dtypes helper function, which causes most Ivy and frontend API tests to be nondeterministic. A straightforward fix would be to always sort any list/tuple converted from a set that is used in a composite strategy. Otherwise, deterministic tests would need to be run with the PYTHONHASHSEED environment variable set.

Steps to Reproduce Bug

  1. Run a (generally) failing Ivy test using the Hypothesis derandomize option
  2. Run the same test again, and it will likely fail with a different example

Environment

Arch Linux using Emacs and venv

Ivy Version

commit 0437358

Backend

  • NumPy
  • TensorFlow
  • PyTorch
  • JAX

Device

No response

@jacksondm33 jacksondm33 added the Bug Report Report bugs detected in Ivy. label Apr 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Report Report bugs detected in Ivy.
Projects
None yet
Development

No branches or pull requests

1 participant