Data races in Argo Rollouts unit tests

[jgwest]

Checklist:

• I've included steps to reproduce the bug.
• I've included the version of argo rollouts.

Describe the bug

A number of data races have crept into the Argo Rollouts unit tests.

I've opened a PR which goes into details of the individual fixes

You can reproduce the issue with:

git clone [email protected]:argoproj/argo-rollouts
go test -race `go list ./... | grep -v ./test/cmd/metrics-plugin-sample`

I'll spare the full test log, but TL;DR:

--- FAIL: TestFailedToCreateProviderError (0.22s)
--- FAIL: TestWatchBlueGreenRollout (10.27s)
--- FAIL: TestRolloutControllerCallback (1.10s)
--- FAIL: TestExperimentControllerCallback (1.10s)
--- FAIL: TestCanaryRolloutScaleDownOldRs (0.24s)
--- FAIL: TestCanaryRolloutScaleDownOldRsDontScaleDownTooMuch (0.23s)
--- FAIL: TestCanaryDontScaleDownOldRsDuringInterruptedUpdate (0.24s)
--- FAIL: TestCanaryScaleDownOldRsDuringInterruptedUpdate (0.25s)
--- FAIL: TestSyncRolloutIgnoreWaitOutsideOfReconciliationPeriod (0.22s)
--- FAIL: TestReconcileOldReplicaSet (0.65s)
	--- FAIL: TestReconcileOldReplicaSet/No_pods_to_scale_down (0.21s)
	--- FAIL: TestReconcileOldReplicaSet/Clean_up_unhealthy_pods (0.22s)
--- FAIL: TestRolloutWithExperimentStep (0.48s)
	--- FAIL: TestRolloutWithExperimentStep/Experiment_Pending_-_no_WeightDestination_created (0.24s)
--- FAIL: TestReconciler_SetWeight (5.44s)
	--- FAIL: TestReconciler_SetWeight/SetWeight (0.00s)
		--- FAIL: TestReconciler_SetWeight/SetWeight/will_return_error_if_base_mapping_not_found (0.13s)
		--- FAIL: TestReconciler_SetWeight/SetWeight/will_create_multiple_canary_mappings_when_provided_multiple_base_mappings (0.13s)
		--- FAIL: TestReconciler_SetWeight/SetWeight/will_return_errors_from_all_mapping_creations (0.13s)
		--- FAIL: TestReconciler_SetWeight/SetWeight/will_return_error_if_base_mapping_defines_the_weight (0.13s)
		--- FAIL: TestReconciler_SetWeight/SetWeight/will_cleanup_canary_mapping_if_current_weight_is_zero (5.13s)
--- FAIL: TestSetHeaderRoute (0.01s)
	--- FAIL: TestSetHeaderRoute/SetHeaderRouteExprs (0.02s)
	--- FAIL: TestSetHeaderRoute/SetHeaderUpdateError (0.02s)
	--- FAIL: TestSetHeaderRoute/SetHeaderRoutePriorityWithNew#01 (0.02s)
--- FAIL: TestSetWeight (0.00s)
	--- FAIL: TestSetWeight/TraefikUpdateError (0.00s)
	--- FAIL: TestSetWeight/SetWeightWithErrorManifest (0.00s)
	--- FAIL: TestSetWeight/SetWeightWithErrorCanaryName (0.00s)
	--- FAIL: TestSetWeight/SetWeightWithErrorStableName (0.00s)
	--- FAIL: TestSetWeight/SetWeight (0.00s)