Skip to content

Commit

Permalink
Flaky test caused by "_created_time" cloning race condition (#2307)
Browse files Browse the repository at this point in the history
Summary:
Pull Request resolved: #2307

This equality check is failing sometimes in Python 3.9 on open source ( T184131441 ):
```
def test_clone_with(self) -> None:
        experiment = get_experiment()
        cloned_experiment = experiment.clone_with()
        self.assertEqual(cloned_experiment.name, "cloned_experiment_" + experiment.name)
        cloned_experiment._name = experiment.name
        self.assertEqual(cloned_experiment, experiment)
```
The error is
```
Experiment(test) (type <class 'ax.core.experiment.Experiment'>) != Experiment(test) (type <class 'ax.core.experiment.Experiment'>).

Fields with different values:

1) _time_created:
2024-04-01 05:05:13.000613 (type <class 'datetime.datetime'>)
!= 2024-04-01 05:05:12.998793 (type <class 'datetime.datetime'>).
```

This is a race condition, where if "get_experiment" and "clone_with" execute quickly the _time_created field will be equal, but if there is delay they'll be unequal.

Approach: reset the time of the cloned experiment to that of the original

Reviewed By: esantorella

Differential Revision: D55596348

fbshipit-source-id: a4ab76df266fb779370c4b4679c21c2712064c06
  • Loading branch information
mgrange1998 authored and facebook-github-bot committed Apr 1, 2024
1 parent 241583b commit 95bafb0
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions ax/core/tests/test_experiment.py
Original file line number Diff line number Diff line change
Expand Up @@ -1126,6 +1126,10 @@ def test_clone_with(self) -> None:
cloned_experiment = experiment.clone_with()
self.assertEqual(cloned_experiment.name, "cloned_experiment_" + experiment.name)
cloned_experiment._name = experiment.name

# the clone_experiment._time_created field is set as datetime.now().
# for it to be equal we need to update it to match experiment.
cloned_experiment._time_created = experiment._time_created
self.assertEqual(cloned_experiment, experiment)

def test_metric_summary_df(self) -> None:
Expand Down

0 comments on commit 95bafb0

Please sign in to comment.