Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support user-supplied profiles #390

Merged
merged 33 commits into from
Jul 26, 2023
Merged

Support user-supplied profiles #390

merged 33 commits into from
Jul 26, 2023

Conversation

jlaneve
Copy link
Collaborator

@jlaneve jlaneve commented Jul 25, 2023

Description

This PR extends the ProfileConfig interface to support two modes:

  1. the profile mapping, which takes an Airflow connection and translates it to a dbt profile
  2. a user-supplied profile file

When using a profile mapping, you can do one of the following:

  1. Use the AutomaticProfileMapping class to auto-select a profile mapping
ProfileConfig(
    profile_name="default",
    target_name="dev",
    profile_mapping=get_automatic_profile_mapping(
        conn_id="airflow_db",
        args={"schema": "public"},
    ),
)
  1. Specify a specific profile mapping
ProfileConfig(
    profile_name="default",
    target_name="dev",
    profile_=PostgresUserPasswordProfileMapping(
        conn_id="airflow_db",
        profile_args={"schema": "public"},
    ),
)

When using your own profile, you can supply the path_to_profiles_yml:

ProfileConfig(
    profile_name="default",
    target_name="dev",
    path_to_profiles_yml=DBT_ROOT_PATH / "jaffle_shop" / "profiles.yml",
)

Related Issue(s)

closes #336
closes #269

Breaking Change?

This changes the ProfileConfig class, which isn't yet user-facing.

Checklist

  • I have made corresponding changes to the documentation (if required)
  • I have added tests that prove my fix is effective or that my feature works

@jlaneve jlaneve requested a review from a team as a code owner July 25, 2023 03:49
@jlaneve jlaneve requested a review from a team July 25, 2023 03:49
Base automatically changed from config-interface to main July 25, 2023 19:53
@netlify
Copy link

netlify bot commented Jul 25, 2023

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name Link
🔨 Latest commit 203c963
🔍 Latest deploy log https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/64c11d3a31add000087f39ea

@codecov
Copy link

codecov bot commented Jul 25, 2023

Codecov Report

Patch coverage: 90.90% and project coverage change: -0.26% ⚠️

Comparison is base (e1ff962) 91.46% compared to head (203c963) 91.21%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #390      +/-   ##
==========================================
- Coverage   91.46%   91.21%   -0.26%     
==========================================
  Files          48       48              
  Lines        1664     1684      +20     
==========================================
+ Hits         1522     1536      +14     
- Misses        142      148       +6     
Files Changed Coverage Δ
cosmos/operators/base.py 97.75% <ø> (ø)
cosmos/config.py 92.50% <81.25%> (-7.50%) ⬇️
cosmos/operators/local.py 78.89% <91.66%> (-2.02%) ⬇️
cosmos/profiles/base.py 94.68% <92.30%> (-0.50%) ⬇️
cosmos/converter.py 97.05% <100.00%> (+0.08%) ⬆️
cosmos/dbt/graph.py 100.00% <100.00%> (ø)
cosmos/profiles/__init__.py 96.00% <100.00%> (+3.14%) ⬆️
cosmos/profiles/snowflake/user_pass.py 96.42% <100.00%> (ø)
cosmos/profiles/snowflake/user_privatekey.py 96.55% <100.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've successfully run basic_cosmos_dag from this branch.

Some comments are in line. Given the timelines, I added "major" to the ones I think are more relevant.

cosmos/config.py Outdated Show resolved Hide resolved
cosmos/config.py Show resolved Hide resolved
dev/dags/cosmos_byo_profile.py Outdated Show resolved Hide resolved
dev/dags/example_virtualenv.py Show resolved Hide resolved
dev/dags/example_virtualenv.py Show resolved Hide resolved
cosmos/config.py Outdated Show resolved Hide resolved
cosmos/config.py Outdated Show resolved Hide resolved
@harels harels merged commit 3185e22 into main Jul 26, 2023
37 of 39 checks passed
@harels harels deleted the byo-profile branch July 26, 2023 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Using normal dbt profiles instead of airflow connections Support user-supplied profiles.yml files
3 participants