[BUG] Sporadic single linkage agglomerative clustering pytest fail

[dantegd]

Describe the bug
Sporadic pytest failure found in #4040:

nrows = 1000, ncols = 50, nclusters = 50, k = 5, connectivity = 'pairwise'
…
def test_single_linkage_sklearn_compare(nrows, ncols, nclusters,
                                            k, connectivity):
…
        # Cluster assignments should be exact, even though the actual
        # labels may differ
>       assert(adjusted_rand_score(cuml_agg.labels_, sk_agg.labels_) == 1.0)
E       assert 0.9779533397545083 == 1.0
E         +0.9779533397545083
E         -1.0

Link to log: https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci/job/cuml/job/prb/job/cuml-gpu-test/CUDA=11.2,GPU_LABEL=gpu,OS=ubuntu18.04,PYTHON=3.8/1114/testReport/junit/cuml.test/test_agglomerative/test_single_linkage_sklearn_compare_pairwise_5_50_50_1000_/

Environment details (please complete the following information):

• Environment location: CI
• Linux Distro/Architecture: Ubuntu 18.04
• GPU Model/Driver: [V100 and driver 396.44]
• CUDA: 11.2

[cjnolet]

This popped up again in #4061: https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci/job/cuml/job/prb/job/cuml-gpu-test/CUDA=11.2,GPU_LABEL=gpu,LINUX_VER=ubuntu18.04,PYTHON=3.8/1145/

[github-actions]

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

[wphicks]

Appeared again on #4761: https://gpuci.gpuopenanalytics.com/job/rapidsai/job/gpuci/job/cuml/job/prb/job/cuml-gpu-test/CUDA=11.5,GPU_LABEL=driver-495,LINUX_VER=ubuntu20.04,PYTHON=3.9/2401/testReport/junit/cuml.tests/test_agglomerative/test_single_linkage_sklearn_compare_pairwise_3_50_50_1000_/