[AIRFLOW-5705] Add secrets backend and support for AWS SSM #6376

dstandish · 2019-10-20T04:07:39Z

Currently we can get connections either in (1) env vars or (2) metastore.

This change provides framework for adding arbitrary creds servers.

As a first example I have implemented AWS SSM Parameter Store.

Getting creds loaded into a new cluster, or syncing for all your developers, can be a bit of a nuisance. With this change it’s possible to just grant your developers access to a SSM param store prefix, and they could immediately have connections in their local airflow setup, rather than having to load them into the database, or store them in a text file. (edited)

Issue link: AIRFLOW-5705

Description above provides context of the change
Commit message/PR title starts with [AIRFLOW-NNNN]. AIRFLOW-NNNN = JIRA ID^*
Unit tests coverage for changes (not needed for documentation changes)
Commits follow "How to write a good git commit message"
Relevant documentation is updated including usage instructions.
I will engage committers as explained in Contribution Workflow Example.

^* For document-only changes commit message can start with [AIRFLOW-XXXX].

In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.
Read the Pull Request Guidelines for more information.

mik-laj · 2019-10-20T04:13:43Z

Work on another key encryption mechanism is available here.
https://github.com/jakahn/incubator-airflow/pulls

dstandish · 2019-10-20T05:00:40Z

@mik-laj interesting. It's related but different yeah? Looks like that one is about providing support for alternative to fernet. Is that right? Here I am trying to provide a means to source connections from arbitrary creds server, i.e. other than env vars / metastore.

airflow/creds.py

mik-laj · 2019-10-20T22:32:05Z

How do you configure connections in other services? Will the web interface work?

dstandish · 2019-10-21T03:26:43Z

Will the web interface work?

No airflow web UI configurability. Currently we can provide connections with env vars and those cannot be changed with web UI either. It would be the same here, when using a alternative creds provider other than the metastore. If you want web UI configurability, just use the metastore. Mind you, your alt provider may have it's own web UI (as is the case with AWS SSM), it just would not be airflow's web UI.

How do you configure connections in other services?

However you want! The point here is to provide a way to override default mechanisms of creds provision. Currently it's env vars and metastore. With this change, you have flexibility. Kindof like hostname_callable; use whatever function you need to use. Just return List[Connection] given a conn_id.

We can provide some built-in standard creds providers (e.g. using AWS SSM). And in this case we would document how to set up. But the main intention is to refactor just a tiny bit to allow for users to override to suit their needs.

codecov-io · 2019-10-22T07:10:55Z

Codecov Report

Merging #6376 into master will decrease coverage by 0.27%.
The diff coverage is 88.17%.

@@            Coverage Diff             @@
##           master    #6376      +/-   ##
==========================================
- Coverage   86.98%   86.71%   -0.28%     
==========================================
  Files         906      910       +4     
  Lines       43855    43923      +68     
==========================================
- Hits        38148    38086      -62     
- Misses       5707     5837     +130

Impacted Files	Coverage Δ
airflow/providers/amazon/aws/secrets/ssm.py	`66.66% <66.66%> (ø)`
airflow/secrets/__init__.py	`94.87% <94.87%> (ø)`
airflow/hooks/base_hook.py	`86.66% <100.00%> (-5.50%)`	⬇️
airflow/secrets/environment_variables.py	`100.00% <100.00%> (ø)`
airflow/secrets/metastore.py	`100.00% <100.00%> (ø)`
airflow/kubernetes/volume_mount.py	`44.44% <0.00%> (-55.56%)`	⬇️
airflow/kubernetes/volume.py	`52.94% <0.00%> (-47.06%)`	⬇️
airflow/kubernetes/pod_launcher.py	`47.18% <0.00%> (-45.08%)`	⬇️
...viders/cncf/kubernetes/operators/kubernetes_pod.py	`69.69% <0.00%> (-25.26%)`	⬇️
airflow/kubernetes/refresh_config.py	`50.98% <0.00%> (-23.53%)`	⬇️
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7f6e6b9...79ca25b. Read the comment docs.

ashb · 2019-10-25T14:32:09Z

I agree with Daniel's view - managing the creds in an external service is out of scope of Airflow. The main reason I see you'd want a feature like this is if you already have creds in an external service and would like Airflow to manage them.

If I was deploying this my Airflow wouldn't even have permission to write/update the creds as it shouldn't ever be doing that.

mik-laj · 2019-10-25T14:50:57Z

@ashb I am mainly worried about backwards compatibility. Some operators are already modifying the list of connections
https://github.com/apache/airflow/blob/master/airflow/gcp/hooks/cloud_sql.py#L938-L952
https://github.com/apache/airflow/blob/master/airflow/gcp/operators/cloud_sql.py#L838-L847
These operators will no longer work with other backends

As for the UI, it seems to me that if it does not work at some settings then it should be turned off in these situations.

ashb · 2019-10-25T15:15:19Z

@ashb I am mainly worried about backwards compatibility. Some operators are already modifying the list of connections
https://github.com/apache/airflow/blob/master/airflow/gcp/hooks/cloud_sql.py#L938-L952
https://github.com/apache/airflow/blob/master/airflow/gcp/operators/cloud_sql.py#L838-L847
These operators will no longer work with other backends

That fells like entirely the wrong place to touch connections in the DB. What's the actual usecase for this? Why does the operator need to write the connection object to the DB? Why not change the hook to take a connection property somehow instead of writing and then deleting the connection.

As for the UI, it seems to me that if it does not work at some settings then it should be turned off in these situations.

Yes, that probably seems fair (or make it read only if that's possible/sensible)

mik-laj · 2019-10-25T15:36:07Z

What's the actual usecase for this? Why does the operator need to write the connection object to the DB?

This Hook retrieves authorization data for an external database based on the instance name in GCP, and then you create a database connection so that hook (PostgresSqlHook / MySqlHook) can read it. Hook does not accept authorization data passed in any other way than via connection. We wanted to avoid entering an ambiguous constructor who accepted authorization data. This could discourage users from using connections instead of creating hooks from the credentials data.

It is worth noting that for the database connection to work it is necessary to run SQL proxy, and in the connection configuration a unique random port number and the name of the local socket are saved

Yes, that probably seems fair (or make it read only if that's possible/sensible)

In this case, he needs an additional method in our backend. Not only the get method, but also the list method.

ashb · 2019-10-25T15:39:38Z

Can you pass a Connection object to the hook(s)? (rather than having the operator write it then the hook read it.)

mik-laj · 2019-10-25T15:52:42Z

Currently, PostgresSQL and MySQL hooks do not support this syntax. If we want we can add such parameters in the constructor, but we should do it before accepting this change. We should also look for others to refer to the connection in the remaining code to avoid similar problems.

I wonder if it's worth separating the Connection object from the Connection database entity. This object is an SQLAlchemy object, but very often it will be used without SQL Alchemy. This can be confusing.

dstandish · 2019-10-27T04:54:18Z

@mik-laj @ashb i have created PR #6440 adding optional param connection to mysql and postgres hooks and altering gcp cloud sql hook / operator to not persist the ephemeral connection to database.

stale · 2019-12-11T05:24:50Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

dstandish · 2019-12-15T00:00:02Z

@mik-laj @ashb what do you think about this one?

now that #6440 (cloud sql dynamic conn generation fix) is merged, there's nothing really standing in the way of this one, except for considerations of whether this is right structure.

Some example use cases...

temporary dev / test cluster:
with astronomer, it's easy to spin up new clusters. so for testing something, or special backfill job we can easily create temporary dev cluster. with this change, we would not have to worry about getting connections into cluster. with one env var, or appropriate IAM, the connection info would be available.
sharing creds with developers
i assume it is common for developers to have connections store in a file like connections.env.sh and then sourced in .bashrc. Maybe they also have to have them in .env file if they also use docker. maybe other folks manage airflow connections -a scripts.
this change could eliminate all of that by centralizing creds storage and allowing developers to pull from the same source. depending on the service, creds could potentially be versioned. creds changes would be synchronized for everyone.

do you think this needs an AIP? if so can i have access to create (apache jira username is dstandish)?

do you have any feedback on current structure? i understand that exactly how we do this is probably more controversial than the general idea of opening up creds sourcing.

airflow/creds/aws_ssm.py

…y call to get_connection make pass config json to backend class init directly

dstandish · 2020-03-14T08:11:21Z

also rebase to latest master. Pydruid fix is merged

done

kaxil

Awesome 🚀

kaxil · 2020-03-14T11:49:04Z

Thanks @dstandish for adding this feature 🎉

potiuk · 2020-03-14T12:15:54Z

Coool!

ashb · 2020-03-16T09:45:45Z

Ummm, why did we merge this without voting on the AIP?

potiuk · 2020-03-16T10:01:41Z

I think we discussed (in the thread on the devlist) that this change did not need voting - especially that it is fully backwards compatible, has literally no change to the current behaviour and has no "generic" secret implementation.

But yes I think it was a bit rushed (my fault, sorry for that) - I think we might want to vote on it retroactively including the option of adding a general secret backend and possibly backporting this to 1.10 ?) (@dstandish ? )

dstandish · 2020-03-16T10:07:45Z

Obvs I defer to you guys on voting.

By "general secret backend" I think you mean arbitrary get secret method, i.e. in addition to get connection. Personally I am not convinced of the need / value of this, but welcome your sales pitch :)

potiuk · 2020-03-16T10:11:28Z

Yeah. Happy to make the pitch @dstandish -> understand you are not convinced.

I think it would be great to start voting on the currently merged scope - even if it is merged now. We can always revert it :)

ashb · 2020-03-16T10:12:11Z

I agree the change isn't breaking etc, but if we have an AIP opened for it we should have voted on it before merging anything -- otherwise there was no point opening an AIP.

kaxil · 2020-03-16T10:15:55Z

I agree the change isn't breaking etc, but if we have an AIP opened for it we should have voted on it before merging anything -- otherwise there was no point opening an AIP.

Apologies I merged it and agree we should have not till we have 3 "+1"s. We already had 2 "+1"s from me & Jarek but I should have waited for the 3rd one. I haven't closed the AIP or marked it as completed yet, so we can bump the thread again and if we don't secure enough votes or someone opposing we could revert this PR.

dstandish · 2020-03-16T10:16:05Z

Do I call for the vote or do you? If me, will do tomorrow

kaxil · 2020-03-16T10:18:58Z

Do I call for the vote or do you? If me, will do tomorrow

I just did :)

potiuk · 2020-03-16T10:27:27Z

Will be the fastest merged AIP ever (-3 days).

kaxil · 2020-03-17T10:41:12Z

tests/providers/amazon/aws/secrets/test_ssm.py

+@mock.patch("airflow.providers.amazon.aws.secrets.ssm.AwsSsmSecretsBackend.get_conn_uri")
+def test_aws_ssm_get_connections(mock_get_uri):
+    mock_get_uri.side_effect = ["scheme://user:pass@host:100"]
+    conn_list = AwsSsmSecretsBackend().get_connections("fake_conn")
+    conn = conn_list[0]
+    assert conn.host == 'host'


This test has an incorrect side_effect. The Mock object AwsSsmSecretsBackend.get_conn_uri returns a string and not a list.

Raised a PR to fix that: #7745

dstandish force-pushed the feature/add-creds-backend branch from d8c9d47 to c4b3d6e Compare October 20, 2019 17:54

mik-laj reviewed Oct 20, 2019

View reviewed changes

airflow/creds.py Outdated Show resolved Hide resolved

dstandish force-pushed the feature/add-creds-backend branch 2 times, most recently from 450f0f2 to b5be7b5 Compare October 21, 2019 04:51

dstandish force-pushed the feature/add-creds-backend branch 3 times, most recently from 8706224 to 5097154 Compare October 25, 2019 08:21

mik-laj added the provider:amazon-aws AWS/Amazon - related issues label Oct 25, 2019

dstandish mentioned this pull request Oct 29, 2019

[AIRFLOW-5768] GCP cloud sql don't store ephemeral connection in db #6440

Merged

5 tasks

stale bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Dec 11, 2019

dstandish force-pushed the feature/add-creds-backend branch from 5097154 to 2caf6a9 Compare December 14, 2019 05:44

stale bot removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Dec 14, 2019

dstandish force-pushed the feature/add-creds-backend branch from 2caf6a9 to e0529d8 Compare December 14, 2019 05:51

ashb reviewed Dec 16, 2019

View reviewed changes

airflow/creds/aws_ssm.py Outdated Show resolved Hide resolved

ashb reviewed Dec 16, 2019

View reviewed changes

airflow/creds/aws_ssm.py Outdated Show resolved Hide resolved

ashb reviewed Dec 16, 2019

View reviewed changes

airflow/creds/aws_ssm.py Outdated Show resolved Hide resolved

dstandish added 4 commits March 14, 2020 01:10

rejigger so that we don't have to re-import backend modules with ever…

b95b637

…y call to get_connection make pass config json to backend class init directly

tighten up documentation

fe94c15

updated config param naming & docs enhancements

1221934

fix config param name: backend_name --> backend

67e2ea3

dstandish force-pushed the feature/add-creds-backend branch from a615382 to 67e2ea3 Compare March 14, 2020 08:10

kaxil approved these changes Mar 14, 2020

View reviewed changes

kaxil merged commit e31e9dd into apache:master Mar 14, 2020

dstandish deleted the feature/add-creds-backend branch March 14, 2020 21:52

This was referenced Mar 16, 2020

[AIRFLOW-5705] Fix bug in Secrets Backend #7742

Merged

[AIRFLOW-5705] Fix bugs in AWS SSM Secrets Backend #7745

Merged

kaxil reviewed Mar 17, 2020

View reviewed changes

kaxil pushed a commit that referenced this pull request Mar 22, 2020

[AIRFLOW-5705] Add secrets backend and support for AWS SSM (#6376)

112c0a9

kaxil pushed a commit to astronomer/airflow that referenced this pull request Apr 28, 2020

[AIRFLOW-5705] Add secrets backend and support for AWS SSM (apache#6376)

2a82f53

kaxil mentioned this pull request Apr 29, 2020

Bump to v1.10.7+astro.10 (BackPort Secrets Backend) astronomer/airflow#573

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AIRFLOW-5705] Add secrets backend and support for AWS SSM #6376

[AIRFLOW-5705] Add secrets backend and support for AWS SSM #6376

dstandish commented Oct 20, 2019 •

edited by boring-cyborg bot

Loading

mik-laj commented Oct 20, 2019

dstandish commented Oct 20, 2019 •

edited

Loading

mik-laj commented Oct 20, 2019

dstandish commented Oct 21, 2019 •

edited

Loading

codecov-io commented Oct 22, 2019 •

edited

Loading

ashb commented Oct 25, 2019

mik-laj commented Oct 25, 2019

ashb commented Oct 25, 2019

mik-laj commented Oct 25, 2019

ashb commented Oct 25, 2019

mik-laj commented Oct 25, 2019 •

edited

Loading

dstandish commented Oct 27, 2019 •

edited

Loading

stale bot commented Dec 11, 2019

dstandish commented Dec 15, 2019 •

edited

Loading

dstandish commented Mar 14, 2020 •

edited

Loading

kaxil left a comment

kaxil commented Mar 14, 2020

potiuk commented Mar 14, 2020

ashb commented Mar 16, 2020

potiuk commented Mar 16, 2020

dstandish commented Mar 16, 2020 •

edited

Loading

potiuk commented Mar 16, 2020

ashb commented Mar 16, 2020

kaxil commented Mar 16, 2020

dstandish commented Mar 16, 2020

kaxil commented Mar 16, 2020

potiuk commented Mar 16, 2020

kaxil Mar 17, 2020

[AIRFLOW-5705] Add secrets backend and support for AWS SSM #6376

[AIRFLOW-5705] Add secrets backend and support for AWS SSM #6376

Conversation

dstandish commented Oct 20, 2019 • edited by boring-cyborg bot Loading

mik-laj commented Oct 20, 2019

dstandish commented Oct 20, 2019 • edited Loading

mik-laj commented Oct 20, 2019

dstandish commented Oct 21, 2019 • edited Loading

codecov-io commented Oct 22, 2019 • edited Loading

Codecov Report

ashb commented Oct 25, 2019

mik-laj commented Oct 25, 2019

ashb commented Oct 25, 2019

mik-laj commented Oct 25, 2019

ashb commented Oct 25, 2019

mik-laj commented Oct 25, 2019 • edited Loading

dstandish commented Oct 27, 2019 • edited Loading

stale bot commented Dec 11, 2019

dstandish commented Dec 15, 2019 • edited Loading

dstandish commented Mar 14, 2020 • edited Loading

kaxil left a comment

Choose a reason for hiding this comment

kaxil commented Mar 14, 2020

potiuk commented Mar 14, 2020

ashb commented Mar 16, 2020

potiuk commented Mar 16, 2020

dstandish commented Mar 16, 2020 • edited Loading

potiuk commented Mar 16, 2020

ashb commented Mar 16, 2020

kaxil commented Mar 16, 2020

dstandish commented Mar 16, 2020

kaxil commented Mar 16, 2020

potiuk commented Mar 16, 2020

kaxil Mar 17, 2020

Choose a reason for hiding this comment

dstandish commented Oct 20, 2019 •

edited by boring-cyborg bot

Loading

dstandish commented Oct 20, 2019 •

edited

Loading

dstandish commented Oct 21, 2019 •

edited

Loading

codecov-io commented Oct 22, 2019 •

edited

Loading

mik-laj commented Oct 25, 2019 •

edited

Loading

dstandish commented Oct 27, 2019 •

edited

Loading

dstandish commented Dec 15, 2019 •

edited

Loading

dstandish commented Mar 14, 2020 •

edited

Loading

dstandish commented Mar 16, 2020 •

edited

Loading