How to deal with MFA in our auto deployment CI workflow?

[damianavila]

Description

Recently, I introduced automatic deployment of hubs in our AWS clusters, #647.
Particularly, some specific work was performed for the Carbonplan cluster (based on AWS EKS instead of kops), #632.
The auto-deployment work as intended for a few days, ie: https://github.com/2i2c-org/pilot-hubs/runs/3533569179?check_suite_focus=true, but it recently failed with an AccessDenied failure, https://github.com/2i2c-org/pilot-hubs/runs/3692933197?check_suite_focus=true.

@jhamman has communicated to us in private messages about 2FA being enabled since last Tuesday, IIRC.
I was expecting 2FA to access the AWS console, as Joe said, but it seems that is extended to the awscli as well because I can not get the resources as usual and the deployer can not deploy as expected.

I have pinged Joe and he confirmed he wanted to enable 2FA for the AWS console, but not the awscli, so there might be some misconfiguration that needs to be addressed.

If Joe really wants to also have 2FA through the awscli, then we have a bigger problem since MFA and CI automation does not usually play well together... (actually, they do not play at all, IMHO).

Value / benefit

Currently, we can not automatically deploy to the Carbonplan hubs.
In fact, we can not even access manually to them through the awscli without 2FA.

Figuring out this issue will allow us to get back to the previous state where auto-deployment by the CI was working as intended.

Implementation details

No response

Tasks to complete

• Check with @jhamman if he really wants 2FA in the awscli (in addition to the AWS console).
• If the answer is no (as it seems from previous conversations), then figure it out the misconfiguration so we do not have 2FA through the awscli.
• If the answer is yes, have a bigger conversation about how to reconcile MFA with CI automation (if that is even possible...)

Updates

No response

[damianavila]

Btw, this is the error I am getting when I tried locally (which resemble what the deployer is experiencing and linked above):

$ aws eks update-kubeconfig --name=carbonplanhub --region=us-west-2

An error occurred (AccessDeniedException) when calling the DescribeCluster operation: User: arn:aws:iam::631969445205:user/damian is not authorized to perform: eks:DescribeCluster on resource: arn:aws:eks:us-west-2:631969445205:cluster/carbonplanhub with an explicit deny

[choldgraf]

So am I correct that our next action here is to understand from @jhamman (and extrapolate this learning to other hubs) that it's not a problem if we do not have MFA on awscli? And if the answer is that this is fine, then we figure out how to give awscli deploy privileges without MFA?

[jhamman]

I was able to reproduce this behavior and determine the Force_MFA policy described here was reaching beyond the console scope. I've temporarily removed the policy while I sort out the correct policy rules. To be clear, our goal is to enforce MFA for console access, not the cli or other applications of programatic access.

[choldgraf]

OK that's good news :-) I think it would be a lot more challenging if communities wanted to force MFA for CI/CD haha

[damianavila]

Indeed, great news! I have tested it on my side as well and can access (again) the cluster using the awscli.

[damianavila]

To be clear, our goal is to enforce MFA for console access, not the cli or other applications of programatic access.

Btw, it seems others found the same issue: https://stackoverflow.com/questions/28177505/enforce-mfa-for-aws-console-login-but-not-for-api-calls

[damianavila]

Btw, checked manual deployment, and it works as expected:

python3 deployer deploy carbonplan staging
Added new context arn:aws:eks:us-west-2:631969445205:cluster/carbonplanhub to /var/folders/mn/8h0hm_395l31nrtn235h29900000gn/T/tmp5b7g36i6
Running helm upgrade --install --create-namespace --wait --namespace staging staging hub-templates/daskhub -f /var/folders/mn/8h0hm_395l31nrtn235h29900000gn/T/tmpi1havntw -f /var/folders/mn/8h0hm_395l31nrtn235h29900000gn/T/tmprez2ms4b
Release "staging" has been upgraded. Happy Helming!
NAME: staging
LAST DEPLOYED: Tue Sep 28 19:39:29 2021
NAMESPACE: staging
STATUS: deployed
REVISION: 24
TEST SUITE: None
Running hub health check...
Testing locally, do not redirect output
.                                                                                                                                                                                                                                                                      [100%]
1 passed in 205.37s (0:03:25)
Health check succeeded!

[damianavila]

And the automatic deployment worked again as expected: https://github.com/2i2c-org/pilot-hubs/runs/3738348334?check_suite_focus=true

[damianavila]

Do we want to keep this one open for the bigger discussion or just close it?
I would suggest not deal with stuff in advance and close it for now until it is a more concrete discussion. Thoughts?

[choldgraf]

wahoo! @damianavila the main thing I can think of here is to have a short documentation about this somewhere. Maybe in the Hub Engineer's guide?

[damianavila]

wahoo! @damianavila the main thing I can think of here is to have a short documentation about this somewhere. Maybe in the Hub Engineer's guide?

Planning to add some stuff in the existing docs PR 😉 : #717

[yuvipanda]

We deal with this for SMCE now, and have a documented process!