Security question

[afarbos]

I want to use s3 and I see that airflow works well with aws (ec2, s3). But s3 mean that I will have data outside. What do you have for security in airflow?

[artwr]

Hi @glodus,

s3 can be locked pretty tightly so that it is only readable with an access_key and a secret_key and not to the rest of the world (You would have to define the appropriate security policy for the resource (bucket, or subdirectory)). This is the setup we use. Airflow is also equipped to handle IAM roles, if your setup permits.
Let me know if you have more questions about this!

[afarbos]

I was more thinking about what I would put on the airflow server, about the "worklow", "pipeline" or dags. It could be very unfortunate that someone understood it.

Not really related, but still a question: Do you handle LDAP authentification or perhaps flask_login already handle it ?

[mistercrunch]

From our perspective security is almost an all-or-nothing as far as executing pipelines. If you do have access to an Airflow box, you pretty much have full access to all the systems Airflow interfaces with. We spoke internally about using Airflow for more sensitive datasets that not all engineers should have access to, and the solution there would be to setup an alternate Airflow environment for that purpose.

The UI is mostly a read-only UI, and there are different level of access there. It's possible to create a limited access where people can just look at pipeline definitions and how they progress.

The flask_login backdoor should allow you to do pretty much anything in terms of authentication. Our setup has a reverse proxy that takes care of ssl and ldap authentication that squeezes http headers in the request, and our airflow_login module just look for the headers to grant the right level of access. Using flask_login and ldap is probably fairly well documented. A temporary setup can be achieved with privileged users using an ssh tunnel.

[afarbos]

Thanks.