Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py #1417

imesias · 2022-08-19T10:03:48Z

SUMMARY

Fix KeyError when comparing state.
Fixes: #1409

ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME

rds_cluster.py

softwarefactory-project-zuul · 2022-08-19T10:25:37Z

Build succeeded.

ansibullbot · 2022-08-19T10:37:24Z

cc @alinabuzachis @jillr @markuman @s-hertel @tremble
click here for bot help

markuman · 2022-08-19T10:42:26Z

@imesias thank you for your PR.
Can you expand the integration test tests/integration/targets/rds_cluster with the case of your origin issue?
Another thing what every PR needs, is a changelog fragment

imesias · 2022-08-19T11:15:56Z

@markuman I've added the changelog fragment and I guess the below should be a sufficient test, problem is there is no db cluster parameter group in the collection to create a cluster parameter group. I could use the cli as a workaround. Please let me know if that will work.

#... some task before to create the parameter group ...
  - name: Modify DB cluster parameter group
    rds_cluster:
      id: "{{ cluster_id }}"
      state: present
      db_cluster_parameter_group_name: "{{ new_db_parameter_group_name }}"
    register: _result_modify_db_parameter_group_name

  - assert:
      that:
        - _result_modify_db_parameter_group_name.changed
        - "'allocated_storage' in _result_modify_db_parameter_group_name"
        - _result_modify_db_parameter_group_name.allocated_storage == 1
        - "'cluster_create_time' in _result_modify_db_parameter_group_name"
        - _result_modify_db_parameter_group_name.copy_tags_to_snapshot == false
        - "'db_cluster_arn' in _result_modify_db_parameter_group_name"
        - "_result_modify_db_parameter_group_name.db_cluster_identifier == '{{ cluster_id }}'"
        - "'db_cluster_parameter_group' in _result_modify_db_parameter_group_name"
        - "'db_cluster_resource_id' in _result_modify_db_parameter_group_name"
        - "'endpoint' in _result_modify_db_parameter_group_name"
        - "'engine' in _result_modify_db_parameter_group_name"
        - _result_modify_db_parameter_group_name.engine == "{{ engine }}"
        - "'engine_mode' in _result_modify_db_parameter_group_name"
        - _result_modify_db_parameter_group_name.engine_mode == "provisioned"
        - "'engine_version' in _result_modify_db_parameter_group_name"
        - "'master_username' in _result_modify_db_parameter_group_name"
        - _result_modify_db_parameter_group_name.master_username == "{{ username }}"
        - "'port' in _result_modify_db_parameter_group_name"
        - _result_modify_db_parameter_group_name.db_cluster_parameter_group == "{{ new_db_parameter_group_name }}"
        - "'status' in _result_modify_db_parameter_group_name"
        - _result_modify_db_parameter_group_name.status == "available"
        - "'tags' in _result_modify_db_parameter_group_name"
        - "'vpc_security_groups' in _result_modify_db_parameter_group_name"

softwarefactory-project-zuul · 2022-08-19T11:32:28Z

Build succeeded.

markuman · 2022-08-19T12:33:01Z

@imesias hm, thereretically yes. like https://github.com/ansible-collections/amazon.aws/blob/ad7a0eb22a3e108451959eceff4f1bd78ea8be7c/tests/integration/targets/ec2_instance/roles/ec2_instance/tasks/termination_protection.yml#L47-L54
But it's a changing aws call in this case, and we must take care that the ressources are also removed when the test is finished (succeeded or failed).
What do you think @alinabuzachis?

imesias · 2022-08-19T19:50:03Z

@markuman @alinabuzachis Expanded integration tests for parameter group test case. Uses cli to create the parameter group and a cleanup task added as well.

alinabuzachis · 2022-08-19T19:54:58Z

@imesias hm, thereretically yes. like https://github.com/ansible-collections/amazon.aws/blob/ad7a0eb22a3e108451959eceff4f1bd78ea8be7c/tests/integration/targets/ec2_instance/roles/ec2_instance/tasks/termination_protection.yml#L47-L54
But it's a changing aws call in this case, and we must take care that the ressources are also removed when the test is finished (succeeded or failed).
What do you think @alinabuzachis?

Agree. If I find some time I’ll work on adding this new module.

alinabuzachis

@imesias thank you for working on this. LGTM!

softwarefactory-project-zuul · 2022-08-19T20:10:14Z

Build failed.

imesias · 2022-08-19T21:29:37Z

@markuman @alinabuzachis hmmm, Seems there are 3 issues.

It looks like I'll need to add engine to the playbook module args for that modify task.

TASK [rds_cluster : Modify DB cluster parameter group] *************************
task path: /home/zuul/.ansible/collections/ansible_collections/community/aws/tests/integration/targets/rds_cluster/roles/rds_cluster/tasks/test_modify.yml:179
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_rds_cluster_payload_f_en1hek/ansible_rds_cluster_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/rds.py", line 206, in call_method
    result = AWSRetry.jittered_backoff(catch_extra_error_codes=retry_codes)(method)(**parameters)
  File "/tmp/ansible_rds_cluster_payload_f_en1hek/ansible_rds_cluster_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 119, in _retry_wrapper
    return _retry_func(
  File "/tmp/ansible_rds_cluster_payload_f_en1hek/ansible_rds_cluster_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 69, in _retry_func
    return func()
  File "/tmp/ansible_rds_cluster_payload_f_en1hek/ansible_rds_cluster_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/modules.py", line 329, in deciding_wrapper
    return unwrapped(*args, **kwargs)
  File "/home/zuul/venv/lib/python3.9/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/zuul/venv/lib/python3.9/site-packages/botocore/client.py", line 677, in _make_api_call
    request_dict = self._convert_to_request_dict(
  File "/home/zuul/venv/lib/python3.9/site-packages/botocore/client.py", line 725, in _convert_to_request_dict
    request_dict = self._serializer.serialize_to_request(
  File "/home/zuul/venv/lib/python3.9/site-packages/botocore/validate.py", line 319, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Missing required parameter in input: "Engine"

I will need to also look at why this conditional in the playbook is evaluating to false in the CI environment.

TASK [rds_cluster : Create DB cluster parameter group if not exists] ***********
task path: /home/zuul/.ansible/collections/ansible_collections/community/aws/tests/integration/targets/rds_cluster/roles/rds_cluster/tasks/test_modify.yml:168
skipping: [modify] => {
    "changed": false,
    "skip_reason": "Conditional result was False"
}

It appears the attached role does not have permission to remove/cleanup a db cluster parameter group.

"stderr": "\nAn error occurred (AccessDenied) when calling the DeleteDBClusterParameterGroup operation: User: arn:aws:sts::XXXXXXXXXXX:assumed-role/ansible-core-ci-test-prod/prod=remote=zuul-cloud is not authorized to perform: rds:DeleteDBClusterParameterGroup on resource: arn:aws:rds:us-east-1:XXXXXXXXX:cluster-pg:test-db-parameter-group-2688144c4346-new because no identity-based policy allows the rds:DeleteDBClusterParameterGroup action"

I will have a look at resolving the first two, not sure who I can talk to about the number 3.

Thanks!

softwarefactory-project-zuul · 2022-08-22T07:56:49Z

Build failed.

imesias · 2022-08-22T08:22:33Z

Hey @markuman @alinabuzachis, The CI AWS credentials require permission to create and remove a DBClusterParameterGroup can you please add permission rds:CreateDBClusterParameterGroup and rds:DeleteDBClusterParameterGroup to the appropriate role to allow the expanded integration test to succeed.

Kind Regards,

softwarefactory-project-zuul · 2022-08-28T07:45:11Z

Build failed.

softwarefactory-project-zuul · 2022-09-02T16:46:38Z

Build succeeded.

imesias · 2022-09-05T07:45:53Z

@gravesm Thanks for the assist and update. Much appreciated. @alinabuzachis @markuman integ tests are now passing.

softwarefactory-project-zuul · 2022-09-05T08:52:10Z

Build succeeded (gate pipeline).

patchback · 2022-09-05T08:52:14Z

Backport to stable-3: 💚 backport PR created

✅ Backport PR branch: patchback/backports/stable-3/b3bc68931de04fb2d8558b01c45988710d24cec3/pr-1417

Backported as #1432

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

….py (#1417) Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py SUMMARY Fix KeyError when comparing state. Fixes: #1409 ISSUE TYPE Bugfix Pull Request COMPONENT NAME rds_cluster.py Reviewed-by: Alina Buzachis <None> Reviewed-by: Markus Bergholz <[email protected]> (cherry picked from commit b3bc689)

patchback · 2022-09-05T08:52:26Z

Backport to stable-4: 💚 backport PR created

✅ Backport PR branch: patchback/backports/stable-4/b3bc68931de04fb2d8558b01c45988710d24cec3/pr-1417

Backported as #1433

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

….py (#1417) Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py SUMMARY Fix KeyError when comparing state. Fixes: #1409 ISSUE TYPE Bugfix Pull Request COMPONENT NAME rds_cluster.py Reviewed-by: Alina Buzachis <None> Reviewed-by: Markus Bergholz <[email protected]> (cherry picked from commit b3bc689)

….py (#1417) (#1433) [PR #1417/b3bc6893 backport][stable-4] Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py This is a backport of PR #1417 as merged into main (b3bc689). SUMMARY Fix KeyError when comparing state. Fixes: #1409 ISSUE TYPE Bugfix Pull Request COMPONENT NAME rds_cluster.py Reviewed-by: Mark Chappell <None>

….py (ansible-collections#1417) Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py SUMMARY Fix KeyError when comparing state. Fixes: ansible-collections#1409 ISSUE TYPE Bugfix Pull Request COMPONENT NAME rds_cluster.py Reviewed-by: Alina Buzachis <None> Reviewed-by: Markus Bergholz <[email protected]> This commit was initially merged in https://github.com/ansible-collections/community.aws See: ansible-collections@b3bc689

imesias changed the title ~~[WIP] Fix DBClusterParameterGroupName Key Error when comparing state~~ [WIP] Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py Aug 19, 2022

imesias mentioned this pull request Aug 19, 2022

community.aws.rds_cluster module fails to manage cluster after creating cluster #1409

Closed

1 task

ansibullbot added WIP Work in progress bug This issue/PR relates to a bug has_issue module module needs_triage new_contributor Help guide this first time contributor plugins plugin (any type) small_patch Hopefully easy to review labels Aug 19, 2022

imesias changed the title ~~[WIP] Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py~~ Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py Aug 19, 2022

ansibullbot added community_review and removed WIP Work in progress needs_triage labels Aug 19, 2022

alinabuzachis approved these changes Aug 19, 2022

View reviewed changes

ansibullbot added integration tests/integration tests tests and removed small_patch Hopefully easy to review labels Aug 19, 2022

imesias and others added 7 commits August 28, 2022 09:13

Add changelog fragment for bugfix

8a9952c

Expand integration tests for parameter group case.

4a703fa

Fix module args for CI job, remove quotes from when clause

ae7c20f

Update parameter group name for consistency

b49c047

Use new_cluster_id as prior test renamed the cluster

fb5751f

Fix module args for CI job, remove quotes from when clause

4bbebed

Use new_cluster_id as prior test renamed the cluster

c5b0a70

imesias force-pushed the im-bugfix-rds-cluster-param-grp branch from fbc1118 to c5b0a70 Compare August 28, 2022 07:20

ansibullbot added community_review and removed merge_commit This PR contains at least one merge commit. Please resolve! needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html labels Aug 28, 2022

Use correct cluster id

837b8b3

markuman added backport-3 PR should be backported to the stable-3 branch backport-4 PR should be backported to the stable-4 branch labels Sep 5, 2022

markuman approved these changes Sep 5, 2022

View reviewed changes

markuman added the mergeit Merge the PR (SoftwareFactory) label Sep 5, 2022

softwarefactory-project-zuul bot merged commit b3bc689 into ansible-collections:main Sep 5, 2022

patchback bot mentioned this pull request Sep 5, 2022

[PR #1417/b3bc6893 backport][stable-3] Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py #1432

Closed

patchback bot mentioned this pull request Sep 5, 2022

[PR #1417/b3bc6893 backport][stable-4] Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py #1433

Merged

renovate bot mentioned this pull request Oct 6, 2022

chore(deps): update dependency community.aws to v5 linxside/ansible-ee#58

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py #1417

Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py #1417

imesias commented Aug 19, 2022

softwarefactory-project-zuul bot commented Aug 19, 2022

ansibullbot commented Aug 19, 2022

markuman commented Aug 19, 2022

imesias commented Aug 19, 2022 •

edited

Loading

softwarefactory-project-zuul bot commented Aug 19, 2022

markuman commented Aug 19, 2022

imesias commented Aug 19, 2022

alinabuzachis commented Aug 19, 2022

alinabuzachis left a comment

softwarefactory-project-zuul bot commented Aug 19, 2022

imesias commented Aug 19, 2022

softwarefactory-project-zuul bot commented Aug 22, 2022

imesias commented Aug 22, 2022 •

edited

Loading

softwarefactory-project-zuul bot commented Aug 28, 2022

softwarefactory-project-zuul bot commented Sep 2, 2022

imesias commented Sep 5, 2022

softwarefactory-project-zuul bot commented Sep 5, 2022

patchback bot commented Sep 5, 2022 •

edited

Loading

patchback bot commented Sep 5, 2022 •

edited

Loading

Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py #1417

Fix KeyError when Cluster Parameter Group is specified in rds_cluster.py #1417

Conversation

imesias commented Aug 19, 2022

SUMMARY

ISSUE TYPE

COMPONENT NAME

softwarefactory-project-zuul bot commented Aug 19, 2022

ansibullbot commented Aug 19, 2022

markuman commented Aug 19, 2022

imesias commented Aug 19, 2022 • edited Loading

softwarefactory-project-zuul bot commented Aug 19, 2022

markuman commented Aug 19, 2022

imesias commented Aug 19, 2022

alinabuzachis commented Aug 19, 2022

alinabuzachis left a comment

Choose a reason for hiding this comment

softwarefactory-project-zuul bot commented Aug 19, 2022

imesias commented Aug 19, 2022

softwarefactory-project-zuul bot commented Aug 22, 2022

imesias commented Aug 22, 2022 • edited Loading

softwarefactory-project-zuul bot commented Aug 28, 2022

softwarefactory-project-zuul bot commented Sep 2, 2022

imesias commented Sep 5, 2022

softwarefactory-project-zuul bot commented Sep 5, 2022

patchback bot commented Sep 5, 2022 • edited Loading

Backport to stable-3: 💚 backport PR created

patchback bot commented Sep 5, 2022 • edited Loading

Backport to stable-4: 💚 backport PR created

imesias commented Aug 19, 2022 •

edited

Loading

imesias commented Aug 22, 2022 •

edited

Loading

patchback bot commented Sep 5, 2022 •

edited

Loading

patchback bot commented Sep 5, 2022 •

edited

Loading