You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Some test data generated via Ivy's helper strategies is nondeterministic (between runs) when using the derandomize option with Hypothesis. Specifically, this appears to be caused by the conversion of sets of string values to lists/tuples, which results in a nondeterministic ordering. This behavior is ultimately caused by Python's hashing randomization:
By default, the hash() values of str and bytes objects are “salted” with an unpredictable random value. Although they remain constant within an individual Python process, they are not predictable between repeated invocations of Python.
This most notably occurs in the get_dtypes helper function, which causes most Ivy and frontend API tests to be nondeterministic. A straightforward fix would be to always sort any list/tuple converted from a set that is used in a composite strategy. Otherwise, deterministic tests would need to be run with the PYTHONHASHSEED environment variable set.
Steps to Reproduce Bug
Run a (generally) failing Ivy test using the Hypothesis derandomize option
Run the same test again, and it will likely fail with a different example
Bug Explanation
Some test data generated via Ivy's helper strategies is nondeterministic (between runs) when using the derandomize option with Hypothesis. Specifically, this appears to be caused by the conversion of sets of string values to lists/tuples, which results in a nondeterministic ordering. This behavior is ultimately caused by Python's hashing randomization:
This most notably occurs in the
get_dtypes
helper function, which causes most Ivy and frontend API tests to be nondeterministic. A straightforward fix would be to always sort any list/tuple converted from a set that is used in a composite strategy. Otherwise, deterministic tests would need to be run with thePYTHONHASHSEED
environment variable set.Steps to Reproduce Bug
Environment
Arch Linux using Emacs and venv
Ivy Version
commit 0437358
Backend
Device
No response
The text was updated successfully, but these errors were encountered: