Add pod level annotations

[nitinatgh]

Added in description to the README.md
Bumped up the chart version of the controller manager
Bumped up the dependency version

Added a new annotation and left the existing one incase it breaks any other previous changes. But not sure if the previous one created was done correctly, so have left it intact.

What this PR does / why we need it: To have annotations set at the pod level for the kubefed controller. Currently it's only restricted to the deployment level.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1447

Special notes for your reviewer: I have created a new annotation as the previous one committed seems to be incorrect and I didn't want to have any breaking changes related to it.

I have tested the changes in a local kind cluster, below is the output from running the helm upgrades

Added the following in the values.yaml

...
  imagePullSecrets: []
  # - name: secretName

  controller:
    annotations: {}
    podAnnotations:
      ad.datadoghq.com/controller-manager.check_names: |
        ["openmetrics"]
      ad.datadoghq.com/controller-manager.init_configs: |
        [{}]
      ad.datadoghq.com/controller-manager.instances: |
        [
          {
            "prometheus_url": "http://%%host%%:%%port%%/metrics",
            "namespace": "kube-federation-system",
            "metrics": [ "*" ],
            "send_distribution_buckets": true,
            "send_distribution_counts_as_monotonic": true,
            "send_histograms_buckets": true,
            "send_monotonic_counter": true
          }
        ]
    replicaCount: 2
    repository: quay.io/kubernetes-multicluster
    image: kubefed
...

helm upgrade dry-run

helm --namespace kube-federation-system upgrade -i kubefed ./kubefed --create-namespace --dry-run

...
# Source: kubefed/charts/controllermanager/templates/deployments.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubefed-controller-manager
  namespace: kube-federation-system
  labels:
    kubefed-control-plane: controller-manager
spec:
  replicas: 2
  selector:
    matchLabels:
      kubefed-control-plane: controller-manager
  strategy: {}
  template:
    metadata:
      labels:
        kubefed-control-plane: controller-manager
      annotations:
        ad.datadoghq.com/controller-manager.check_names: |
          ["openmetrics"]
        ad.datadoghq.com/controller-manager.init_configs: |
          [{}]
        ad.datadoghq.com/controller-manager.instances: |
          [
            {
              "prometheus_url": "http://%%host%%:%%port%%/metrics",
              "namespace": "kube-federation-system",
              "metrics": [ "*" ],
              "send_distribution_buckets": true,
              "send_distribution_counts_as_monotonic": true,
              "send_histograms_buckets": true,
              "send_monotonic_counter": true
            }
          ]
    spec:
      nodeSelector:
        {}
      tolerations:
...

helm upgrade

$ kubectl get pods -A

NAMESPACE                NAME                                             READY   STATUS    RESTARTS   AGE
kube-federation-system   kubefed-admission-webhook-67fb48d5f7-hhjhm       1/1     Running   0          23h
kube-federation-system   kubefed-controller-manager-76c659dd6b-924tp      1/1     Running   0          11s
kube-federation-system   kubefed-controller-manager-76c659dd6b-rq27r      1/1     Running   0          13s
kube-system              coredns-558bd4d5db-g29mw                         1/1     Running   0          24h
kube-system              coredns-558bd4d5db-rfggp                         1/1     Running   0          24h
kube-system              etcd-cluster1-control-plane                      1/1     Running   0          24h

$ kubectl describe pod -n kube-federation-system  kubefed-controller-manager-76c659dd6b-924tp

Name:         kubefed-controller-manager-76c659dd6b-924tp
Namespace:    kube-federation-system
Priority:     0
Node:         cluster1-control-plane/172.18.0.2
Start Time:   Wed, 25 Aug 2021 13:55:07 +0100
Labels:       kubefed-control-plane=controller-manager
              pod-template-hash=76c659dd6b
Annotations:  ad.datadoghq.com/controller-manager.check_names: ["openmetrics"]
              ad.datadoghq.com/controller-manager.init_configs: [{}]
              ad.datadoghq.com/controller-manager.instances:
                [
                  {
                    "prometheus_url": "http://%%host%%:%%port%%/metrics",
                    "namespace": "kube-federation-system",
                    "metrics": [ "*" ],
                    "send_distribution_buckets": true,
                    "send_distribution_counts_as_monotonic": true,
                    "send_histograms_buckets": true,
                    "send_monotonic_counter": true
                  }
                ]
              kubectl.kubernetes.io/restartedAt: 2021-08-25T12:55:04Z
Status:       Running
IP:           10.244.0.64
IPs:
  IP:           10.244.0.64
...

Also would like to give a special thanks to https://github.com/kiich for helping me out on this change!

Thanks

Nitin

[nitinatgh]

This PR replaces #1465 due local complications during commits
@yuswift
/assign @makkes
/assign @xunpan
Apologies again for this.

[makkes]

/lgtm

[nitinatgh]

/assign @irfanurrehman
/assign @xunpan

[nitinatgh]

/retest

[k8s-ci-robot]

@nitinatgh: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[irfanurrehman]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: irfanurrehman, makkes, nitinatgh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [irfanurrehman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment