IAM Roles for Service Accounts (IRSA) causes terminating loop

[by-nelson]

Describe the bug
When adding service account annotation for AWS IRSA the jenkins pod remains in a terminating state.

To Reproduce
Add service account annotation with AWS Role ARN:

jenkins:
...
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: "arn:aws:iam::account-id:role/role-name"
...

Deploy the jenkins manifest updated:

kubectl apply -f jenkins.yaml

Restart the jenkins pod so that the AWS environment variables are added:

kubectl delete pod jenkins

After this jenkins pod loops in a terminating state

NAME                                        READY   STATUS        RESTARTS   AGE
jenkins                                     0/2     Terminating   0          4s

Additional information

Kubernetes version: v1.22.17-eks
Jenkins Operator version: v0.8.1

Describe jenkins output

Name:                      jenkins-jenkins-operated                                                                                                                                                                
Namespace:                 jenkins                                                                                                                                                                        
Priority:                  0                                                                                                                                                                                       
Service Account:           jenkins-operator-jenkins-operated                                                                                                                                                       
Node:                      ip-10-x-x-x.us-west-x.compute.internal/10.x.x.x                                                                                                                                   
Start Time:                Tue, 06 Aug 2024 11:41:28 -0600                                                                                                                                                         
Labels:                    app=jenkins-operator                                                                                                                                                                    
                           jenkins-cr=jenkins-operated                                                                                                                                                             
Annotations:               kubernetes.io/psp: eks.privileged                                                                                                                                                       
Status:                    Terminating (lasts <invalid>)                                                                                                                                                           
Termination Grace Period:  30s                                                                                                                                                                                     
IP:                                                                                                                                                                                                                
IPs:                       <none>                                                                                                                                                                                  
Controlled By:             Jenkins/jenkins-operated                                                                                                                                                                
Containers:                                                                                                                                                                                                        
  jenkins-master:                                                                                                                                                                                                  
    Container ID:                                                                                                                                                                                                  
    Image:         jenkins/jenkins:2.452.3-lts-jdk21                                                                                                                    
    Image ID:                                                                                                                                                                                                      
    Ports:         8080/TCP, 50000/TCP                                                                                                                                                                             
    Host Ports:    0/TCP, 0/TCP                                                                                                                                                                                    
    Command:                                                                                                                                                                                                       
      bash                                                                                                                                                                                                         
      -c                                                                                                                                                                                                           
      /var/jenkins/scripts/init.sh && exec /usr/bin/tini -s -- /usr/local/bin/jenkins.sh                                                                                                                           
    State:          Waiting                                                                                                                                                                                        
      Reason:       ContainerCreating                                                                                                                                                                              
    Ready:          False                                                                                                                                                                                          
    Restart Count:  0                                                                                                                                                                                              
    Limits:                                                                                                                                                                                                        
      cpu:     1500m                                                                                                                                                                                               
      memory:  3Gi                                                                                                                                                                                                 
    Requests:                                                                                                                                                                                                      
      cpu:      1                                                                                                                                                                                                  
      memory:   500Mi                                                                                                                                                                                              
    Liveness:   http-get http://:http/login delay=100s timeout=5s period=10s #success=1 #failure=12                                                                                                                
    Readiness:  http-get http://:http/login delay=80s timeout=1s period=10s #success=1 #failure=10                                                                                                                 
    Environment:                                                                                                                                                                                                   
      COPY_REFERENCE_FILE_LOG:      /var/lib/jenkins/copy_reference_file.log                                                                                                                                       
      SECRETS:                      /var/jenkins/configuration-as-code-secrets                                                                                                                                     
      JAVA_OPTS:                    -XX:MinRAMPercentage=50.0 -XX:MaxRAMPercentage=80.0 -Djenkins.install.runSetupWizard=false -Djava.awt.headless=true                                                            
      JENKINS_HOME:                 /var/lib/jenkins                                                                                                                                                               
      AWS_STS_REGIONAL_ENDPOINTS:   regional                                                                                                                                                                       
      AWS_DEFAULT_REGION:           us-west-x                                                                                                                                                                      
      AWS_REGION:                   us-west-x                                                                                                                                                                      
      AWS_ROLE_ARN:                 arn:aws:iam::account-id:role/role-name                                                                                                                     
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token

Operator logs

operator.log

[brokenpip3]

yes unfortunately it's a well know issue, it's ok for you if I close this one in favor of #586?

Any PR to address this are welcome :)

[by-nelson]

Sure, I didn't see this was already on the radar.

I will try to take a look at it and let you know if I can work on a PR for it, thanks!