Mask credentials in Rendered Template tab of task instance details page

[abhilash1in]

Apache Airflow version: 1.10.12

Kubernetes version (if you are using kubernetes) (use kubectl version): N/A

Environment:

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): Ubuntu 18.04.4 LTS
• Kernel (e.g. uname -a): Linux upcairflow01 4.15.0-115-generic #116-Ubuntu SMP Wed Aug 26 14:04:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
• Others:

What happened:
I have some credentials stored on Airflow which I'm passing to a PythonOperator via op_kwargs as follows:

from airflow.hooks.base_hook import BaseHook
from airflow.operators.python_operator import PythonOperator
from sample_module import sample_python_callable

CONN_ID = 'sample_conn_id'
conn = BaseHook.get_connection(CONN_ID)

# other DAG related code
# ...
sample_task = PythonOperator(
    task_id='sample_task',
    python_callable=sample_python_callable,
    op_kwargs={
        'credentials': {
            'host': conn.host,
            'username': conn.login,
            'password': conn.password
        }
    }
)
# ...
# other DAG related code

The actual plaintext credentials are exposed in the Rendered Template tab of the task instance details page.

What you expected to happen:

Expected all credentials/passwords stored as Airflow Connections to be masked in the UI.

There should be a way for Airflow to identify credentials/passwords and mask them everywhere (UI and logs) automatically.

How to reproduce it:
Create a task using PythonOperator similar to the example above and pass some credentials stored as Airflow Connections. They are visible in plaintext on the UI.

Anything else we need to know:

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[mik-laj]

You should pass conn_id to callable and this callable should use conn = BaseHook.get_connection(CONN_ID) to read credentials. This is the recommended and official way to manage credentials data for operators..

Otherwise, your DAG will make a lot of database queries. In the community, we have automatic tests on CIs that detect and block such DAG file.

airflow/tests/test_example_dags.py

Line 47 in ff72327

def test_should_not_do_database_queries(self):

[kaxil]

Similar issues:

• Hide sensitive data in UI #8421
• Airflow Logging Sensitive Information #9638

[abhilash1in]

@mik-laj @kaxil thank you, the similar issues linked above covers this in more detail. Closing this issue.