-
Notifications
You must be signed in to change notification settings - Fork 159
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BigQuery keyfile_dict profile mapping use env vars for sensitive fields #471
BigQuery keyfile_dict profile mapping use env vars for sensitive fields #471
Conversation
👷 Deploy Preview for amazing-pothos-a3bca0 processing.
|
Codecov ReportPatch coverage has no change and project coverage change:
Additional details and impacted files@@ Coverage Diff @@
## main #471 +/- ##
==========================================
- Coverage 91.52% 91.11% -0.41%
==========================================
Files 50 50
Lines 1770 1768 -2
==========================================
- Hits 1620 1611 -9
- Misses 150 157 +7
☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The code coverage report was being shown incorrectly for PRs derived from forks, as seen in the PR #471 Incorrect parts of the code were highlighted: ![Screenshot 2023-08-16 at 11 25 27](https://github.com/astronomer/astronomer-cosmos/assets/272048/b657e7a2-4c35-4869-a99d-56d4ddc3c823) Reference: https://app.codecov.io/gh/astronomer/astronomer-cosmos/pull/471?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomer#diff-Y29zbW9zL3Byb2ZpbGVzL2JpZ3F1ZXJ5L3NlcnZpY2VfYWNjb3VudF9rZXlmaWxlX2RpY3QucHk=
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you very much for addressing the feedback, @jbandoro , and fixing this issue.
Feature (pending documentation!) * Support dbt global flags (via dbt_cmd_global_flags in `operator_args` by @tatiana in #469 Enhancements * Hide sensitive field when using BigQuery keyfile_dict profile mapping by @jbandoro in #471 Bug fixes * Fix bug on select node add exclude selector subset ids logic by @jensenity in #463 * Refactor dbt ls to run from a temporary directory, to avoid Read-only file system errors during DAG parsing, by @tatiana in #414 Others * Docs: Fix RenderConfig load argument by @jbandoro in #466 * Enable CI integration tests from external forks by @tatiana in #458 * Improve CI tests runtime by @tatiana in #457 * Change CI to run coverage after tests pass by @tatiana in #461 * Fix forks code revision in code coverage by @tatiana in #472 * [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #467" i
**Features** * Support dbt global flags (via dbt_cmd_global_flags in operator_args) by @tatiana in #469 * Support parsing DAGs when there are no connections by @jlaneve in #489 **Enhancements** * Hide sensitive field when using BigQuery keyfile_dict profile mapping by @jbandoro in #471 * Consistent Airflow Dataset URIs, inlets and outlets with `Openlineage package <https://pypi.org/project/openlineage-integration-common/>`_ by @tatiana in #485. `Read more <https://astronomer.github.io/astronomer-cosmos/configuration/lineage.html>`_. * Refactor ``LoadMethod.DBT_LS`` to run from a temporary directory with symbolic links by @tatiana in #488 * Run ``dbt deps`` when using ``LoadMethod.DBT_LS`` by @DanMawdsleyBA in #481 * Update Cosmos log color to purple by @harels in #494 * Change operators to log ``dbt`` commands output as opposed to recording to XCom by @tatiana in #513 **Bug fixes** * Fix bug on select node add exclude selector subset ids logic by @jensenity in #463 * Refactor dbt ls to run from a temporary directory, to avoid Read-only file system errors during DAG parsing, by @tatiana in #414 * Fix profile_config arg in DbtKubernetesBaseOperator by @david-mag in #505 * Fix SnowflakePrivateKeyPemProfileMapping private_key reference by @nacpacheco in #501 * Fix incorrect temporary directory creation in VirtualenvOperator init by @tatiana in #500 * Fix log propagation issue by @tatiana in #498 * Fix PostgresUserPasswordProfileMapping to retrieve port from connection by @jlneve in #511 **Others** * Docs: Fix RenderConfig load argument by @jbandoro in #466 * Enable CI integration tests from external forks by @tatiana in #458 * Improve CI tests runtime by @tatiana in #457 * Change CI to run coverage after tests pass by @tatiana in #461 * Fix forks code revision in code coverage by @tatiana in #472 * [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #467 * Drop support to Python 3.7 in the CI test matrix by @harels in #490 * Add Airflow 2.7 to the CI test matrix by @tatiana in #487 * Add MyPy type checks to CI since we exceeded pre-commit disk quota usage by @tatiana in #510
Description
I noticed that if
GoogleCloudServiceAccountDictProfileMapping
is used for profile mapping, the full keyfile json is logged in the UI. See example below:This PR hides the sensitive private key fields (
private_key
andprivate_key_id
) as env vars so that they will not show up in the logs, see below:Would appreciate feedback since unlike other profiles, the
secret_fields
here are nested within thekeyfile_json
field so I had to overrideBaseProfileMapping.env_vars
here to make it work.Related Issue(s)
None
Breaking Change?
None
Checklist