Streamline multicluster checks for per-cluster service mirror controllers

[adleong]

As described in linkerd/rfc#31 we are moving to a model where we will not install the service-mirror-controller as part of linkerd multicluster install but instead create a service mirror controller per linked target cluster as part of linkerd multicluster link. This allows us to streamline and simplify some complex and conditional logic around source and target cluster checks in linkerd check.

The multicluster checks in linkerd check include category of source cluster checks and a category of target cluster checks. Each of these categories can be run or skipped, depending on the values of the SourceCluster, TargetCluster, and Multicluster flags. The conditions which influence these options, the effect of the option, and the way they interact is non-obvious.

Under the new model, these checks can be reorganized to simplify the conditional logic:

If the Link CRD exists or if --multicluster is passed explicitly then the multicluster addon is assumed to have been installed and we should check:

• That the Link CRD exists
• That all cluster scoped RBAC needed for service mirror controllers exists (ClusterRoles and ClusterRoleBindings)

For each Link resource that exists:

• Check that the Link resource is valid and that the credentials can be used to connect to the target cluster API
• Check that the specified gateway in the target cluster is alive
• Check that the namespace scoped RBAC for the associated service mirror controller exists (ServiceAccount, Role, RoleBinding)
• Check that the associated service mirror controller is running
• Check that the trust roots match
• Check that the gateway mirror exists, has endpoints, and is healthy

For each mirror service that exists:

• Check that it has endpoints
• Check that it has an associated Link resource (i.e. that it is not orphaned)
• Check that it is not exported (no daisy chains)

Notice that we have removed the target cluster checks. This is a consequence of shifting the ownership of mirroring onto the source cluster. Target clusters no longer need to annotate their gateway in any special way and so the concept of a "target cluster gateway" is meaningless outside of the context of a source cluster linking to it.

[adleong]

@zaharidichev I would love to get your feedback on the above ^^

[adleong]

Fixed by #4710