Refactor the Makefile and hack/ scripts

[ConnorJC3]

Is this a bug fix or adding new feature?

"New Feature"/Refactoring

What is this PR about? / Why do we need it?

This PR contains a significant refactor of the Makefile and e2e testing scripts. Its goals are:

• Primary Goal: Modularize run.sh to enable separate cluster creation, test running, and cluster deletion. Today, running E2E tests locally requires waiting on the creation and termination of a cluster before and after the run, which takes significant time. Developers should be able to run E2E tests locally against a pre-existing cluster, as well as easily create and delete clusters for testing that follow the standard E2E setup.
• Primary Goal: Document the available Makefile targets and how to use them. New contributors should not need to go digging in the Makefile to discover how to perform simple tasks such as building the driver or running e2e tests.
• Secondary Goal: Centralize dependency management. We should have a single place that all local tools are installed - rather than the existing code which has multiple different methods of achieving the same task of installing a binary from GitHub.
• Secondary Goal: Simply Makefile - many of the existing makefile targets (in particular, the e2e targets) are long, unwieldy, and hard to reason about. Clean up the makefile and hack/ directory as much as possible to remove unneeded complexity.
• Explicit Non-Goal: Significant changes to how the image is built or how the e2e tests are ran. I do think we should look into improvements in both of these places in the future, but this PR already contains enough changes as is. This PR should not change the behavior of our CI (other than making it run faster and easier to develop for). To this end, eksctl.sh, kops.sh, and metrics.sh are largely or entirely unmodified.
• Explicit Non-Goal: Changes to Go files - this PR is only about refactoring the testing infrastructure, not the tests themselves.

What testing is done?

Significant local testing as well as CI, see the new docs/makefile.md for the available makefile targets to test against locally

[github-actions]

Code Coverage Diff

This PR does not change the code coverage

[AndrewSirenko]

Still working through this PR, but I found this visualization helpful

Pre-PR:

Post-PR:

Generated with vak/makefile2dot: Visualize your Makefile using GraphViz dot utility

[AndrewSirenko]

On a high-level, these are great primary and secondary goals, and from a quick-look, the makefile (and file changes at large) are cleaner and more discoverable. I especially appreciate the README as well as the separated commands (e.g e2e/windows, verify/...)

Can we rebase this PR to separate linting, code extraction, filename changes, and variable name changes into separate commits from the main changes to business logic?

Perhaps commits in this order:

1. Variable name changes
2. Code extraction
3. Filename changes
4. Main Business logic commit (Bonus points if there are separate commits for Each primary/secondary goal)
5. Linting

Perhaps by running git reset master followed by re-committing changes by hunk/File?

Let me know if parallel programming committing would help in this endeavor. I did something similar for the original versions of #1846 and #1845 before squashing those commits.

Also:

Significant local testing as well as CI, see the new docs/makefile.md for the available makefile targets to test against locally

Means that you tested each makefile target locally and on CI, correct?

[ConnorJC3]

Can we rebase this PR to separate linting, code extraction, filename changes, and variable name changes into separate commits from the main changes to business logic?

Is there a significant advantage to doing the large amount work to split this into separate commits? They will ultimately need to be squashed for merge (as all of the commits you listed will be in a broken, partial state) and doing so greatly increases the effort required to update the PR for review comments.

Means that you tested each makefile target locally and on CI, correct?

All the targets were tested locally, and CI runs the targets it already does (make verify, make test, and all the e2e tests)

[torredil]

Hitting a quota limit here:

2023-12-11 21:30:31 [✖]  creating OIDC provider: operation error IAM: CreateOpenIDConnectProvider, https response error StatusCode: 409, RequestID: ce25e2a1-fa71-48e4-93a9-0e26ffd23971, LimitExceeded: Cannot exceed quota for OpenIdConnectProvidersPerAccount: 100

cc: @dims

[dims]

@torredil cleaned up a few, please try again?

[torredil]

/retest

[AndrewSirenko]

Great job on this!! Super beneficial to onboarding contributors to the project, as well as maintaining maintainer sanity. Ran a few of the commands, will make sure to run through all of them when I find more time. Will start using this makefile for daily driver development and see what else I find.

[AndrewSirenko]

LGTM once rebased.

[torredil]

/lgtm
/approve
/hold

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: torredil

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [torredil]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ConnorJC3]

/remove-hold

[ConnorJC3]

/retest