Add support for env vars in RenderConfig for dbt ls parsing

[jbandoro]

Description

Currently there is a workaround to have environment variables that are required when parsing a dbt project with the dbt ls load mode by setting them with os.environ in the DAG file.

This is what is currently done in the cosmos dev dag here since that env var is required for parsing with dbt ls. The problem with setting os.environ in that python file is that for the sqlite integration test it was enabling this test to unexpectedly pass (which also requires that env var).

This PR adds support for env_vars as an argument for RenderConfig and sets/unsets the environment variables in a context manager for the dbt ls graph parsing.

Related Issue(s)

closes #550 and closes #646

Breaking Change?

None

Checklist

• I have made corresponding changes to the documentation (if required)
• I have added tests that prove my fix is effective or that my feature works

[netlify]

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name	Link
🔨 Latest commit	06c4b80
🔍 Latest deploy log	https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/655e4aa9f97ec50008714605

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (24aa38e) 92.93% compared to head (cb136c2) 92.97%.

❗ Current head cb136c2 differs from pull request most recent head 06c4b80. Consider uploading reports for the commit 06c4b80 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #690      +/-   ##
==========================================
+ Coverage   92.93%   92.97%   +0.04%     
==========================================
  Files          55       55              
  Lines        2292     2306      +14     
==========================================
+ Hits         2130     2144      +14     
  Misses        162      162              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tatiana]

Hey @jbandoro this change looks great, thank you!

Moving forward, we need to find a way so users don't need to set this type of information both in the RenderConfig and in the ExecutionConfig - since, in most cases, they will want the same environment vars defined in both places. A related recent request was to set dbt --vars in a way they are reused both in RenderConfig and ExecutionConfig.

Would ProjectConfig be a more suitable place for this? Or should we have a new config object, altogether?

@jlaneve would love your thoughts on having a unified interface as well

[jbandoro]

Moving forward, we need to find a way so users don't need to set this type of information both in the RenderConfig and in the ExecutionConfig - since, in most cases, they will want the same environment vars defined in both places. A related recent request was to set dbt --vars in a way they are reused both in RenderConfig and ExecutionConfig.

Would ProjectConfig be a more suitable place for this? Or should we have a new config object, altogether?

I agree that having to specify env variables twice, once for parsing and again for execution is not ideal, and I can't think of a reason why they would be different? For example in the example cosmos source dev dag we have:

astronomer-cosmos/dev/dags/example_cosmos_sources.py

Lines 61 to 83 in 06c4b80

	render_config = RenderConfig(
	node_converters={
	DbtResourceType("source"): convert_source, # known dbt node type to Cosmos (part of DbtResourceType)
	DbtResourceType("exposure"): convert_exposure, # dbt node type new to Cosmos (will be added to DbtResourceType)
	},
	env_vars={"DBT_SQLITE_PATH": DBT_SQLITE_PATH},
	)
	example_cosmos_sources = DbtDag(
	# dbt/cosmos-specific parameters
	project_config=ProjectConfig(
	DBT_ROOT_PATH / "simple",
	),
	profile_config=profile_config,
	render_config=render_config,
	operator_args={"env": {"DBT_SQLITE_PATH": DBT_SQLITE_PATH}},
	# normal dag parameters
	schedule_interval="@daily",
	start_date=datetime(2023, 1, 1),
	catchup=False,
	dag_id="example_cosmos_sources",
	)

I agree It would be cleaner and less confusing to have env vars specified in one place like ProjectConfig, and deprecate the env in the operator args and the RenderConfig here.