Add machine type upgrades to our recurring internal engineering tasks calendar

[consideRatio]

I propose we schedule the following task recurringly, for example once a year.

Goals

• To levrage modern hardware available in clouds to help communities get the most out of their spent money
• To not use too many different machine types in order to reduce compelxity and uncertainty

Recurring epic's tasks

• For each cloud provider, compare currently used set of machine types and sizees, and recommend if we should change something.
  • AWS
  • GCP
  • Azure
• Pilot use of new machine types and sizes in one cluster per cloud provider
  • AWS
  • GCP
  • Azure
• On successfull pilots, transition to new machine types (time-efficient to do during a k8s upgrade)
  • AWS
  • GCP
  • Azure

Notes

• With new instance types, the memory/cpu capacity could change slightly, so tuning requested memory is relevant
• In AWS, instance types available pods per node is important and could have been impacted. Verify that a new machine type doesn't have less pods per node. If it has more, thats great.
• Related issues:
  • Adopt modern (n2, r5, v5) machines and replace older cost inefficient (n1, r4, v3/v4) #3339
  • Transition to r7i instances from r5 instances #4750