Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decrypt secrets from SystemsManagerParameterStoreBackend #9214

Merged
merged 2 commits into from
Jun 14, 2020

Conversation

n2p5
Copy link

@n2p5 n2p5 commented Jun 10, 2020

This is my first PR, so pardon me if I missed any steps, but this proposed change is also very simple.

The current implementation of SystemsManagerParameterStoreBackend _get_secret uses

            response = self.client.get_parameter(
                Name=ssm_path, WithDecryption=False
            )

but this only allows for secrets to be stored in clear text.

This PR changes the value to WithDecryption=True, which is backwards compatible with clear text values, but also supports KMS decryption in the API call.

for reference in the API docs:
https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_GetParameter.html

WithDecryption
Return decrypted values for secure string parameters. This flag is ignored for String and StringList parameter types.

Type: Boolean

Required: No

After reviewing the test coverage and the documentation, I don't think there are any changes that need to be made. The are no behavior changes for users other than this will support a storage option and will natively support tools like chamber cli tooling

I guess there could be an argument made to update documentation stating this feature is now supported?


Make sure to mark the boxes below before creating PR: [x]

  • Description above provides context of the change
  • Unit tests coverage for changes (not needed for documentation changes)
  • Target Github ISSUE in description if exists
  • Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions.
  • I will engage committers as explained in Contribution Workflow Example.

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

@boring-cyborg boring-cyborg bot added area:secrets provider:amazon-aws AWS/Amazon - related issues labels Jun 10, 2020
@boring-cyborg
Copy link

boring-cyborg bot commented Jun 10, 2020

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:

  • Pay attention to the quality of your code (flake8, pylint and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Be sure to read the Airflow Coding style.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: [email protected]
    Slack: https://apache-airflow-slack.herokuapp.com/

@BasPH
Copy link
Contributor

BasPH commented Jun 10, 2020

How is this change backwards compatible? Does AWS automatically "fallback" in case of a non-secure string?

@robosante
Copy link

Per the boto docs, it's a no-op on non-secure strings: https://docs.aws.amazon.com/goto/WebAPI/ssm-2014-11-06/GetParameter

Parameters
...
WithDecryption (boolean) -- Return decrypted values for secure string parameters. This flag is ignored for String and StringList parameter types.

@n2p5
Copy link
Author

n2p5 commented Jun 10, 2020

@BasPH yes - if you set WithDecryption=True and you are dealing with a String or StringList, instead of a SecureString, it returns the normal output. SecureString has an added reference to a KMS alias used for envelop encryption and SSM will return the clear text of that.

from the reference docs:

This flag is ignored for String and StringList parameter types.
https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_GetParameter.html

@BasPH
Copy link
Contributor

BasPH commented Jun 10, 2020

Nice, looks good. Could you add a test for SecureString to airflow.tests.providers.amazon.aws.secrets.test_systems_manager.py?

@ashb ashb added this to the Airflow 1.10.11 milestone Jun 10, 2020
@kaxil kaxil changed the title flipped boolean to support KMS decryption for ssm ps Decrypt secrets from SystemsManagerParameterStoreBackend Jun 14, 2020
@kaxil kaxil merged commit ffb8574 into apache:master Jun 14, 2020
@boring-cyborg
Copy link

boring-cyborg bot commented Jun 14, 2020

Awesome work, congrats on your first merged pull request!

kaxil pushed a commit that referenced this pull request Jun 22, 2020
kaxil pushed a commit to kaxil/airflow that referenced this pull request Jun 27, 2020
potiuk pushed a commit that referenced this pull request Jun 29, 2020
kaxil pushed a commit that referenced this pull request Jul 1, 2020
cfei18 pushed a commit to cfei18/incubator-airflow that referenced this pull request Mar 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:secrets provider:amazon-aws AWS/Amazon - related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants