Adopt modern (n2, r5, v5) machines and replace older cost inefficient (n1, r4, v3/v4)

[consideRatio]

Here the kinds of modern isntance types I think we should use, examplified with three equivalent highmem nodes. The n2, r5, and v5 machines are all documented to be Intel Xeon, third generation Icy Lake processors, and their cost per performance is lower than the older options (n1, r4, v3/v4).

• GCP machine families and pricing, n2-highmem-4 / n2-highmem-16 / n2-highmem-64
• AWS instance types and pricing, r5.xlarge / r5.4xlarge / r5.16xlarge
• Azure instance types and pricing, Standard_E4s_v5 / Standard_E16s_v5 / Standard_E64s_v5

Action points

• Transition n1 to n2
  • Add inline fixme comments next to wherever old nodes are declared in terraform config to upgrade to n2
  • Await the changes be made alongside other changes made in the infra, such as a k8s upgrade
  • UPDATE: 2024-10-23, only pangeo-hubs, which is down, still make use of n1 for non-gpu situations
• Transition r4 to r5
• Transition v3/v4 to v5
  • Add inline fixme comments next to wherever old nodes are declared in terraform config to upgrade to v5
  • Await the changes be made alongside other changes made in the infra, such as a k8s upgrade

Historical context

• Phasing out use of n1 machines in favor of n2 machines on GKE #2923, closed in favor of this