Use AzureClusterIdentity when running ci e2e tests

[nader-ziada]

What type of PR is this?
/kind feature

What this PR does / why we need it:

• use AzureClusterIdentity in ci to make sure it working in all cases

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

TODOs:

• squashed commits
• includes documentation
• adds unit tests

Release note:

- Update aad-pod-identity to v1.8.0 which upgrades CRDs from apiextensions/v1beta1 to apiextensions/v1
- Update calico version used by templates and tests

[nader-ziada]

/test pull-cluster-api-provider-azure-e2e

[nader-ziada]

/test pull-cluster-api-provider-azure-e2e

[nader-ziada]

/test pull-cluster-api-provider-azure-e2e

[nader-ziada]

the private cluster test specifically is failing every time, looking into it

[nader-ziada]

the private cluster test specifically is failing every time, looking into it

I believe the private cluster test is failing because of this issue here https://azure.github.io/aad-pod-identity/docs/troubleshooting/#token-requests-calls-fail-with-io-timeout

[nader-ziada]

/test pull-cluster-api-provider-azure-e2e-windows

[nader-ziada]

more info about issues with tests failing in this PR can be found in this slack thread https://kubernetes.slack.com/archives/CEX9HENG7/p1621362051002500

tldr;

• the nmi pod is having dns issues and can't authenticate when the management cluster is running on the Azure (not on a kind cluster) as in the case of the private cluster test. The dns issues happen when using dnsPolicy=ClusterFirstWithHostNet as recommended for any pod using hostNetwork=true
• The worker node has no issues connecting and the nmi pod works fine with dnsPolicy=ClusterFirst
• but there seems to be an issue with either how we configure the nmi pod, or the azure cluster vnet

[nader-ziada]

@nader-ziada tried out tilt locally, found two issues:

• if you run tilt up a second time it fails with error: failed to create secret secrets "cluster-identity-secret" already exists: we should delete the secret and recreate it every tilt up so it's idempotent (same thing we do with configmaps)
• the secret env var is getting printed in tilt logs, any way we can avoid that?

• added the delete statement similar to the configmaps
• added the quiet=true option to prevent printing

[devigned]

All nits.

[CecileRobertMichon]

Do we want to keep all 7 commits in the final merge or are some of these squashable?

[nader-ziada]

Do we want to keep all 7 commits in the final merge or are some of these squashable?

I think they are for different parts, but I can squash if you prefer

[CecileRobertMichon]

I think they are for different parts, but I can squash if you prefer

nope, all good, just checking they are distinct

[nader-ziada]

applied @devigned comments about newlines

[devigned]

/lgtm

[CecileRobertMichon]

/lgtm

[nader-ziada]

thanks for all the help with debugging this PR @CecileRobertMichon @devigned @mboersma

[devigned]

@nader-ziada great work on this one! This one was a tough task. Thank you for taking this on.