Make sure that NSM candidate selection is working as fast as possible

[denis-tingaikin]

Overview

Seems like permutation mechanics doesn't work well. See at networkservicemesh/cmd-registry-k8s#376

This could be super useful for folks who want to use their custom registry service or some buggy registry service.

By this reason we need to test and fix our nse iteration logic in the discover and roundrobin chain elements and test it on the real clusteres

Test scenario

1. Deploy NSM
2. Deploy 100 nses
3. Deploy 1 nsc
4. Make nses from step 2 unreachable
5. Deploy a new reachable nse

Expected: nsc should reach the nse
Actual: ?

TODO list

• add unit tests
• fix issues if they exist
• check the scenario on the packet cluster

[NikitaSkrynnik]

Tested basic NSM setup with 10, 20, 40 dead NSEs and 1 working NSE on Azure Cluster. Also tested 40 dead forwarders to check discover in NSMGR. Both discovers in nsmgr and forwarder work fast. Connection is established on the first attepmt.

Here are the logs for each experiment:

1. 10 dead NSEs and 1 working NSE - 10nse.zip
2. 20 dead NSEs and 1 working NSE - 20nse.zip
3. 40 dead NSEs and 1 working NSE - 40nse.zip
4. 40 dead FWDs and 1 workfing NSE - 40fwds.zip
5. 40 dead and deleted NSEs and 1 working NSE - 40nses-deleted.zip

[denis-tingaikin]

Closing by #1438 (comment)