test: add test to validate the controller actions to keep the cluster data

[hectorj2f]

What this PR does / why we need it:

It is a continuation of #1201 which has been stale for a long period. I cherry picked their changes and added some tests.

This test also cover other functions of the kubefedcluster controller.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hectorj2f

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [hectorj2f]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[irfanurrehman]

Recommend: Verify that the data updated (API Endpoint) equals what is updated above?

[hectorj2f]

👍

[irfanurrehman]

@hectorj2f Thanks for doing this. The test itself looks quite allright. A few nits (see comments).

Additionally little bit of history for your reference:
We had the integration tests (the kind that your wrote here) earlier, where a local etcd and api server were used for individual tests; however we saw that the overall time taken to complete all tests against the PRs was crossing 1 hour. At some point we did fold all those integration tests into e2e to save time (currently the test run still takes close to 40+ mins).
The difference I am talking about is:
Test run with etcd and api-server in this PR: 7.507s
Pure unit test from an older build: .021s

Having said that, I do agree that the current test is not verifying the cluster state change and I think its ok to introduce the new test you have added, covering the missing piece as here (It would be worse in terms of time consumed to verify this scenario in e2e).
Can I also recommend that the current unit test be left behind as it is, and add the test that you have introduced separately under test/integration (If you think its easy enough to do)?
Again, really appreciate you doing this.

[hectorj2f]

@irfanurrehman I'd check how this can all be introduced without causing a lot of confusion or messing up the code structure. I certainly realized that there are many pieces of the kubefed's code that are not tested either because it escapes from the e2e radar or it is too complex to create mocks for all these tess.

[hectorj2f]

@irfanurrehman I addressed your comments. I didn't move the new test to test/integration due to the amount of additional changes that will imply.

[irfanurrehman]

Thanks @hectorj2f
/lgtm