ec2_instance doesn't handle AWS API eventual consistency errors

[ajdonnison]

Summary

In a percentage of cases launching a new instance, the play fails with InvalidInstanceID.NotFound exception even after successfully creating the instance. This appears to be due to the eventual consistency model of the AWS API as described in the AWS API Reference Troubleshooting section

Issue Type

Bug Report

Component Name

ec2_instance

Ansible Version

$ ansible --version

ansible [core 2.15.5]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.11/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.11.6 (main, Oct  3 2023, 03:36:49) [GCC 12.2.1 20220924] (/usr/local/bin/python)
  jinja version = 3.1.2
  libyaml = True

Collection Versions

$ ansible-galaxy collection list

# /usr/local/lib/python3.11/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    6.5.0  
ansible.netcommon             5.2.0  
ansible.posix                 1.5.4  
ansible.utils                 2.11.0 
ansible.windows               1.14.0 
arista.eos                    6.1.2  
awx.awx                       22.7.0 
azure.azcollection            1.18.1 
check_point.mgmt              5.1.1  
chocolatey.chocolatey         1.5.1  
cisco.aci                     2.7.0  
cisco.asa                     4.0.2  
cisco.dnac                    6.7.5  
cisco.intersight              1.0.27 
cisco.ios                     4.6.1  
cisco.iosxr                   5.0.3  
cisco.ise                     2.5.16 
cisco.meraki                  2.16.5 
cisco.mso                     2.5.0  
cisco.nso                     1.0.3  
cisco.nxos                    4.4.0  
cisco.ucs                     1.10.0 
cloud.common                  2.1.4  
cloudscale_ch.cloud           2.3.1  
community.aws                 6.3.0  
community.azure               2.0.0  
community.ciscosmb            1.0.6  
community.crypto              2.15.1 
community.digitalocean        1.24.0 
community.dns                 2.6.2  
community.docker              3.4.9  
community.fortios             1.0.0  
community.general             7.5.0  
community.google              1.0.0  
community.grafana             1.5.4  
community.hashi_vault         5.0.0  
community.hrobot              1.8.1  
community.libvirt             1.3.0  
community.mongodb             1.6.3  
community.mysql               3.7.2  
community.network             5.0.0  
community.okd                 2.3.0  
community.postgresql          2.4.3  
community.proxysql            1.5.1  
community.rabbitmq            1.2.3  
community.routeros            2.10.0 
community.sap                 1.0.0  
community.sap_libs            1.4.1  
community.skydive             1.0.0  
community.sops                1.6.6  
community.vmware              3.10.0 
community.windows             1.13.0 
community.zabbix              2.1.0  
containers.podman             1.10.3 
cyberark.conjur               1.2.2  
cyberark.pas                  1.0.23 
dellemc.enterprise_sonic      2.2.0  
dellemc.openmanage            7.6.1  
dellemc.powerflex             1.9.0  
dellemc.unity                 1.7.1  
f5networks.f5_modules         1.26.0 
fortinet.fortimanager         2.2.1  
fortinet.fortios              2.3.2  
frr.frr                       2.0.2  
gluster.gluster               1.0.2  
google.cloud                  1.2.0  
grafana.grafana               2.2.3  
hetzner.hcloud                1.16.0 
hpe.nimble                    1.1.4  
ibm.qradar                    2.1.0  
ibm.spectrum_virtualize       1.12.0 
infinidat.infinibox           1.3.12 
infoblox.nios_modules         1.5.0  
inspur.ispim                  1.3.0  
inspur.sm                     2.3.0  
junipernetworks.junos         5.3.0  
kubernetes.core               2.4.0  
lowlydba.sqlserver            2.2.1  
microsoft.ad                  1.3.0  
netapp.aws                    21.7.0 
netapp.azure                  21.10.0
netapp.cloudmanager           21.22.0
netapp.elementsw              21.7.0 
netapp.ontap                  22.7.0 
netapp.storagegrid            21.11.1
netapp.um_info                21.8.0 
netapp_eseries.santricity     1.4.0  
netbox.netbox                 3.14.0 
ngine_io.cloudstack           2.3.0  
ngine_io.exoscale             1.1.0  
ngine_io.vultr                1.1.3  
openstack.cloud               2.1.0  
openvswitch.openvswitch       2.1.1  
ovirt.ovirt                   3.2.0  
purestorage.flasharray        1.21.0 
purestorage.flashblade        1.14.0 
purestorage.fusion            1.6.0  
sensu.sensu_go                1.14.0 
servicenow.servicenow         1.0.6  
splunk.es                     2.1.0  
t_systems_mms.icinga_director 1.33.1 
telekom_mms.icinga_director   1.34.1 
theforeman.foreman            3.14.0 
vmware.vmware_rest            2.3.1  
vultr.cloud                   1.10.0 
vyos.vyos                     4.1.0  
wti.remote                    1.0.5  

AWS SDK versions

$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.28.66
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /usr/local/lib/python3.11/site-packages
Requires: botocore, jmespath, s3transfer
Required-by: 
---
Name: botocore
Version: 1.31.66
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email: 
License: Apache License 2.0
Location: /usr/local/lib/python3.11/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed
CONFIG_FILE() = None
HOST_KEY_CHECKING(env: ANSIBLE_HOST_KEY_CHECKING) = False

OS / Environment

Ubuntu 22.04

Steps to Reproduce

    - name: launch instance
      delegate_to: 127.0.0.1
      amazon.aws.ec2_instance:
        image_id: "{{ lambda_ami }}"
        region: "{{ aws_region }}"
        key_name: "{{ aws_key_name }}"
        vpc_subnet_id: "{{ subnet_id }}"
        instance_type: "{{ instance_type }}"
        iam_instance_profile: "{{ instance_role }}"
        instance_initiated_shutdown_behavior: terminate
        wait: yes
        wait_timeout: 900
        security_groups: "{{ aws_security_group | list }}"
        count: 1
        state: present
        detailed_monitoring: no
        user_data: "{{ lookup('file', '/tmp/user-data') }}"
        metadata_options:
          instance_metadata_tags: enabled
        tags:
          Name: "{{ lambda_name }}"
          env: "{{ run_environment }}"
          role: "{{ lambda_base }}"
        volumes:
          - device_name: /dev/sda1
            ebs:
              encrypted: yes
              kms_key_id: "{{aws_ebs_cmk_key_id}}"
              volume_size: "{{ aws_root_volume_size }}"
              volume_type: gp3
              delete_on_termination: true
              throughput: "{% if aws_root_volume_size|int >= 250 %}250{% else %}125{% endif %}"
      tags: launch
      register: ec2_info

Run the above a number of times

Expected Results

I expect that when an instance is successfully created, this task does not fail with any errors.

Actual Results

The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_amazon.aws.ec2_instance_payload_h9e9jyak/ansible_amazon.aws.ec2_instance_payload.zip/ansible_collections/amazon/aws/plugins/modules/ec2_instance.py", line 1459, in diff_instance_and_params
  File "/tmp/ansible_amazon.aws.ec2_instance_payload_h9e9jyak/ansible_amazon.aws.ec2_instance_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/retries.py", line 105, in deciding_wrapper
    return retrying_wrapper(*args, **kwargs)
  File "/tmp/ansible_amazon.aws.ec2_instance_payload_h9e9jyak/ansible_amazon.aws.ec2_instance_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 119, in _retry_wrapper
    return _retry_func(
  File "/tmp/ansible_amazon.aws.ec2_instance_payload_h9e9jyak/ansible_amazon.aws.ec2_instance_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 68, in _retry_func
    return func()
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 535, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.10/site-packages/botocore/client.py", line 980, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidInstanceID.NotFound) when calling the DescribeInstanceAttribute operation: The instance ID 'i-0dca1382bb783136c' does not exist

Note that this occurs randomly, approximately 2% of all invocations.

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ajdonnison]

I believe the issue is with:
https://github.com/ansible-collections/amazon.aws/blob/main/plugins/modules/ec2_instance.py#L1457-L1461
and could be in other places.

A possible change would be:

    if params.get("security_group") or params.get("security_groups"):
        try:
            value = AWSRetry.jittered_backoff(
                catch_extra_error_codes=["InvalidInstanceID.NotFound"],
            )(client.describe_instance_attribute)(
                Attribute="groupSet", InstanceId=id_
            )
        except (botocore.exceptions.BotoCoreError, botocore.exceptions.ClientError) as e:
            module.fail_json_aws(e, msg=f"Could not describe attribute groupSet for instance {id_}")

[alinabuzachis]

@ajdonnison Thank you for raising this issue. Can you please update your amazon.aws collection version to 7.2.0 and give it a new try? Please let us know if you're still facing the same issue.

[gravesm]

Closing as this issue should be fixed in the latest version. If this is still a problem, please reopen this issue.