Automate hub deployments on AWS

[choldgraf]

Description

Once we have a cluster running, we need to automate hub deployments on AWS. We also need to configure our already-running hubs to follow this process.

Value / benefit

This will let us automatically add new AWS hubs, and will standardize our infrastructure deployments in general. As AWS is a very popular cloud provider, this is an important step to make us more confident in adding new hubs.

Implementation details

This might be done most easily by automating the deployment of our specific AWS hubs, and then documenting / generalizing how to do it for a generic hub.

Completion criteria

Given an AWS cluster that already exists, we can deploy a new hub (or upgrade an existing hub) on that cluster via editing YAML in our pilot-hubs/ repository.

Tasks to complete

• Automatically deploy carbonplan hub #632
• Use an AWS IAM Role / User to fetch kubernetes credentials in our deployer #381
• Automate deployment of Farallon hubs Use an AWS deployer user to fetch kubernetes credentials in our deployer #673
• Automate deployment of OpenScapes hubs Enable openscapes (and re-enable farallon) deployment by the CI #691
• Add documentation about best-practices in deploying hubs on AWS #696

Updates

• 2021-09-14 - We plan to finish this deliverable during this sprint. It should just be a matter of deploying OpenScapes on this infrastructure, and writing some documentation.
• 2021-09-21 - Our plan is to finish up Enable openscapes (and re-enable farallon) deployment by the CI #691 and then to finish Add documentation about best-practices in deploying hubs on AWS #696 and consider this issue closed!

[damianavila]

The second step depends on #381

[damianavila]

Researched a little bit on farallon and openscapes existing user and groups...
In farallon, we would need to create a new deployer user under the 2i2c-engineers group
In openscapes, we could use the existing 2i2cAdministrator as the deployer user.
I think that is OK because it will showcase how to set up the CI stuff in environments with different granularity on users and groups. In the future, we might consolidate things in a "preferred" pattern.

[damianavila]

The deployer user created in the Farallon space, more details in #381 (comment)
There is also a PR linked in that issue that, hopefully, will close it.

[damianavila]

#691 now deals with openscapes (among other things).

[damianavila]

Re-opened because it was automatically closed and there is still stuff to do here...

[choldgraf]

@damianavila the only thing remaining here is to document our current setup, is that right? If that's the case, are you OK with just creating a new issue for the documentation specifically, tracking it here, and removing this issue from the deliverables backlog?

[damianavila]

@damianavila the only thing remaining here is to document our current setup, is that right?

Correct!

If that's the case, are you OK with just creating a new issue for the documentation specifically, tracking it here,

The issue already exists and it is linked in the first message of the thread.

and removing this issue from the deliverables backlog?

Sure, I have removed it from the deliverables backlog.

[damianavila]

General documentation was merged on #717.
Upgrading existing AWS hubs by CI/CD is currently available.
Deploying the hub for the first time is not fully automated, although it is a semi-automatic process outlined at https://infrastructure.2i2c.org/en/latest/howto/operate/new-hub/aws.html#add-a-hub-to-an-aws-kops-based-cluster