Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Teradata profile throwing error with mock profile #1087

Closed
1 task done
Tracked by #1103
sc250072 opened this issue Jul 10, 2024 · 1 comment · Fixed by #1088
Closed
1 task done
Tracked by #1103

[Bug] Teradata profile throwing error with mock profile #1087

sc250072 opened this issue Jul 10, 2024 · 1 comment · Fixed by #1088
Labels
area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc area:profile Related to ProfileConfig, like Athena, BigQuery, Clickhouse, Spark, Trino, etc bug Something isn't working dbt:list Primarily related to dbt list command or functionality execution:virtualenv Related to Virtualenv execution environment parsing:dbt_ls Issues, questions, or features related to dbt_ls parsing triage-needed Items need to be reviewed / assigned to milestone
Milestone

Comments

@sc250072
Copy link
Contributor

Astronomer Cosmos Version

Other Astronomer Cosmos version (please specify below)

If "Other Astronomer Cosmos version" selected, which one?

1.6.0a1

dbt-core version

1.8.1

Versions of dbt adapters

dbt-teradata 1.8.1

LoadMode

AUTOMATIC

ExecutionMode

VIRTUALENV

InvocationMode

None

airflow version

2.9.2

Operating System

Distributor ID: Ubuntu Description: Ubuntu 22.04.4 LTS Release: 22.04 Codename: jammy

If a you think it's an UI issue, what browsers are you seeing the problem on?

No response

Deployment

Virtualenv installation

Deployment details

Deployed 1.6.0a1 under python3.10 virtual environment

What happened?

cosmos 1.6.0a1 throwing an error with mock profile
raise CosmosLoadDbtException(f"Unable to run {command} due to the error:\n{details}") cosmos.dbt.graph.CosmosLoadDbtException: Unable to run ['/home/airbyte/devtools/cosmos/cosmos-venv/bin/dbt', 'ls', '--output', 'json', '--project-dir', '/tmp/tmp84a8an_e', '--profiles-dir', '/tmp/cosmos/profile/cf3dc9700b875516f605be1e8410b5dc7d9765fe25b7da42140dfc441de87144', '--profile', 'generated_profile', '--target', 'dev'] due to the error: 06:48:04 Running with dbt=1.8.3 06:48:05 Encountered an error: Runtime Error Credentials in profile "generated_profile", target "dev" invalid: Runtime Error Must specifyschema in profile

Relevant log output

standalone | Starting Airflow Standalone
standalone | Checking database is initialized
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
WARNI [airflow.models.crypto] empty cryptography key - values will not be stored encrypted.
standalone | Database ready
/home/airbyte/devtools/cosmos/cosmos-venv/lib/python3.10/site-packages/flask_limiter/extension.py:337 UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
WARNI [airflow.configuration] section/key [openlineage/namespace] not found in config
[2024-07-09T23:36:50.459-0700] {local.py:76} WARNING - (astronomer-cosmos) - To enable emitting Openlineage events, upgrade to Airflow 2.7 or install astronomer-cosmos[openlineage].
WARNI [astronomer-cosmos-cosmos.operators.local] To enable emitting Openlineage events, upgrade to Airflow 2.7 or install astronomer-cosmos[openlineage].
triggerer  | ____________       _____________
triggerer  | ____    |__( )_________  __/__  /________      __
triggerer  | ____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
triggerer  | ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
triggerer  | _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
triggerer  | [2024-07-09 23:36:52 -0700] [43492] [INFO] Starting gunicorn 22.0.0
triggerer  | [2024-07-09 23:36:52 -0700] [43492] [INFO] Listening at: http://[::]:8794 (43492)
triggerer  | [2024-07-09 23:36:52 -0700] [43492] [INFO] Using worker: sync
triggerer  | [2024-07-09 23:36:52 -0700] [43496] [INFO] Booting worker with pid: 43496
triggerer  | [2024-07-09 23:36:52 -0700] [43497] [INFO] Booting worker with pid: 43497
triggerer  | [2024-07-09T23:36:52.186-0700] {configuration.py:1050} WARNING - section/key [openlineage/namespace] not found in config
triggerer  | [2024-07-09T23:36:52.215-0700] {local.py:76} WARNING - (astronomer-cosmos) - To enable emitting Openlineage events, upgrade to Airflow 2.7 or install astronomer-cosmos[openlineage].
triggerer  | [2024-07-09T23:36:52.215-0700] {local.py:76} WARNING - To enable emitting Openlineage events, upgrade to Airflow 2.7 or install astronomer-cosmos[openlineage].
triggerer  | [2024-07-09T23:36:52.399-0700] {triggerer_job_runner.py:174} INFO - Setting up TriggererHandlerWrapper with handler <FileTaskHandler (NOTSET)>
triggerer  | [2024-07-09T23:36:52.400-0700] {triggerer_job_runner.py:230} INFO - Setting up logging queue listener with handlers [<RedirectStdHandler <stdout> (NOTSET)>, <TriggererHandlerWrapper (NOTSET)>]
triggerer  | [2024-07-09T23:36:52.408-0700] {triggerer_job_runner.py:331} INFO - Starting the triggerer
scheduler  | ____________       _____________
scheduler  | ____    |__( )_________  __/__  /________      __
scheduler  | ____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
scheduler  | ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
scheduler  | _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
scheduler  | [2024-07-09T23:36:52.499-0700] {configuration.py:1050} WARNING - section/key [openlineage/namespace] not found in config
scheduler  | [2024-07-09T23:36:52.523-0700] {local.py:76} WARNING - (astronomer-cosmos) - To enable emitting Openlineage events, upgrade to Airflow 2.7 or install astronomer-cosmos[openlineage].
scheduler  | [2024-07-09T23:36:52.523-0700] {local.py:76} WARNING - To enable emitting Openlineage events, upgrade to Airflow 2.7 or install astronomer-cosmos[openlineage].
scheduler  | [2024-07-09T23:36:52.576-0700] {task_context_logger.py:63} INFO - Task context logging is enabled
scheduler  | [2024-07-09T23:36:52.577-0700] {executor_loader.py:235} INFO - Loaded executor: SequentialExecutor
scheduler  | [2024-07-09T23:36:52.614-0700] {scheduler_job_runner.py:796} INFO - Starting the scheduler
scheduler  | [2024-07-09T23:36:52.615-0700] {scheduler_job_runner.py:803} INFO - Processing each file at most -1 times
scheduler  | [2024-07-09T23:36:52.629-0700] {manager.py:170} INFO - Launched DagFileProcessorManager with pid: 43503
scheduler  | [2024-07-09T23:36:52.631-0700] {scheduler_job_runner.py:1595} INFO - Adopting or resetting orphaned tasks for active dag runs
scheduler  | [2024-07-09 23:36:52 -0700] [43502] [INFO] Starting gunicorn 22.0.0
scheduler  | [2024-07-09 23:36:52 -0700] [43502] [INFO] Listening at: http://[::]:8793 (43502)
scheduler  | [2024-07-09 23:36:52 -0700] [43502] [INFO] Using worker: sync
scheduler  | [2024-07-09T23:36:52.640-0700] {settings.py:60} INFO - Configured default timezone UTC
scheduler  | [2024-07-09 23:36:52 -0700] [43504] [INFO] Booting worker with pid: 43504
scheduler  | [2024-07-09T23:36:52.974-0700] {manager.py:393} WARNING - Because we cannot use more than 1 thread (parsing_processes = 2) when using sqlite. So we set parallelism to 1.
scheduler  | [2024-07-09 23:36:52 -0700] [43505] [INFO] Booting worker with pid: 43505
webserver  | /home/airbyte/devtools/cosmos/cosmos-venv/lib/python3.10/site-packages/flask_limiter/extension.py:337 UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
webserver  | [2024-07-09T23:36:54.659-0700] {configuration.py:1050} WARNING - section/key [openlineage/namespace] not found in config
webserver  | [2024-07-09T23:36:54.683-0700] {local.py:76} WARNING - (astronomer-cosmos) - To enable emitting Openlineage events, upgrade to Airflow 2.7 or install astronomer-cosmos[openlineage].
webserver  | [2024-07-09T23:36:54.683-0700] {local.py:76} WARNING - To enable emitting Openlineage events, upgrade to Airflow 2.7 or install astronomer-cosmos[openlineage].
webserver  | [2024-07-09 23:36:55 -0700] [43501] [INFO] Starting gunicorn 22.0.0
webserver  | [2024-07-09 23:36:55 -0700] [43501] [INFO] Listening at: http://0.0.0.0:8088 (43501)
webserver  | [2024-07-09 23:36:55 -0700] [43501] [INFO] Using worker: sync
webserver  | [2024-07-09 23:36:55 -0700] [43546] [INFO] Booting worker with pid: 43546
webserver  | [2024-07-09 23:36:55 -0700] [43565] [INFO] Booting worker with pid: 43565
webserver  | [2024-07-09 23:36:55 -0700] [43566] [INFO] Booting worker with pid: 43566
webserver  | [2024-07-09 23:36:55 -0700] [43569] [INFO] Booting worker with pid: 43569
webserver  | 172.20.45.43 - - [09/Jul/2024:23:36:57 -0700] "GET /home HTTP/1.1" 200 33657 "http://vaa-cdt-ubuntu-5115:8088/connection/edit/54" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
standalone | Airflow is ready
standalone | Login with username: admin  password: VNYV2dWpZ6PcCc7x
standalone | Airflow Standalone is for development purposes only. Do not use this in production!
webserver  | 172.20.45.43 - - [09/Jul/2024:23:37:01 -0700] "GET /connection/list/ HTTP/1.1" 200 136437 "http://vaa-cdt-ubuntu-5115:8088/home" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
webserver  | 172.20.45.43 - - [09/Jul/2024:23:37:05 -0700] "GET /connection/edit/54 HTTP/1.1" 200 34285 "http://vaa-cdt-ubuntu-5115:8088/connection/list/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
webserver  | [2024-07-09T23:37:21.543-0700] {crypto.py:82} WARNING - empty cryptography key - values will not be stored encrypted.
webserver  | [2024-07-09T23:37:21.546-0700] {base.py:84} INFO - Using connection ID 'ZVxNuM3I' for task execution.
webserver  | 172.20.45.43 - - [09/Jul/2024:23:37:21 -0700] "POST /api/v1/connections/test HTTP/1.1" 200 1160 "http://vaa-cdt-ubuntu-5115:8088/connection/edit/54" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
webserver  | [2024-07-09T23:37:32.483-0700] {base.py:84} INFO - Using connection ID 'DrSURsy8' for task execution.
webserver  | [2024-07-09T23:37:33.713-0700] {sql.py:470} INFO - Running statement: select 1, parameters: None
webserver  | [2024-07-09T23:37:33.856-0700] {sql.py:479} INFO - Rows affected: 1
webserver  | 172.20.45.43 - - [09/Jul/2024:23:37:34 -0700] "POST /api/v1/connections/test HTTP/1.1" 200 68 "http://vaa-cdt-ubuntu-5115:8088/connection/edit/54" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
webserver  | [2024-07-09T23:37:36.497-0700] {crypto.py:82} WARNING - empty cryptography key - values will not be stored encrypted.
webserver  | 172.20.45.43 - - [09/Jul/2024:23:37:36 -0700] "POST /connection/edit/54 HTTP/1.1" 302 283 "http://vaa-cdt-ubuntu-5115:8088/connection/edit/54" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
webserver  | 172.20.45.43 - - [09/Jul/2024:23:37:37 -0700] "GET /connection/list/ HTTP/1.1" 200 136643 "http://vaa-cdt-ubuntu-5115:8088/connection/edit/54" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
webserver  | 172.20.45.43 - - [09/Jul/2024:23:37:39 -0700] "GET /home HTTP/1.1" 200 33657 "http://vaa-cdt-ubuntu-5115:8088/connection/list/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
triggerer  | [2024-07-09T23:37:52.483-0700] {triggerer_job_runner.py:481} INFO - 0 triggers currently running
triggerer  | [2024-07-09T23:38:52.542-0700] {triggerer_job_runner.py:481} INFO - 0 triggers currently running
triggerer  | [2024-07-09T23:39:52.597-0700] {triggerer_job_runner.py:481} INFO - 0 triggers currently running
triggerer  | [2024-07-09T23:40:52.653-0700] {triggerer_job_runner.py:481} INFO - 0 triggers currently running
triggerer  | [2024-07-09T23:41:52.722-0700] {triggerer_job_runner.py:481} INFO - 0 triggers currently running
scheduler  | [2024-07-09T23:41:52.909-0700] {scheduler_job_runner.py:1595} INFO - Adopting or resetting orphaned tasks for active dag runs
triggerer  | [2024-07-09T23:42:52.785-0700] {triggerer_job_runner.py:481} INFO - 0 triggers currently running
triggerer  | [2024-07-09T23:43:52.838-0700] {triggerer_job_runner.py:481} INFO - 0 triggers currently running
triggerer  | [2024-07-09T23:44:52.896-0700] {triggerer_job_runner.py:481} INFO - 0 triggers currently running

triggerer  | [2024-07-09T23:45:52.948-0700] {triggerer_job_runner.py:481} INFO - 0 triggers currently running
^Cstandalone | Shutting down components
standalone | Complete
(cosmos-venv) airbyte@vaa-cdt-ubuntu-5115:~/airflow$ export DBT_VIR_PATH=~/devtools/cosmos/cosmos-venv
(cosmos-venv) airbyte@vaa-cdt-ubuntu-5115:~/airflow$ export AIRFLOW_HOME=~/airflow/
(cosmos-venv) airbyte@vaa-cdt-ubuntu-5115:~/airflow$ airflow standalone
standalone | Starting Airflow Standalone
standalone | Checking database is initialized
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
WARNI [airflow.configuration] section/key [openlineage/namespace] not found in config
[2024-07-09T23:48:03.517-0700] {local.py:76} WARNING - (astronomer-cosmos) - To enable emitting Openlineage events, upgrade to Airflow 2.7 or install astronomer-cosmos[openlineage].
WARNI [astronomer-cosmos-cosmos.operators.local] To enable emitting Openlineage events, upgrade to Airflow 2.7 or install astronomer-cosmos[openlineage].
ERROR [airflow.models.dagbag.DagBag] Failed to import: /home/airbyte/airflow/dags/cosmos_profile_mapping_dag.py
Traceback (most recent call last):
  File "/home/airbyte/devtools/cosmos/cosmos-venv/lib/python3.10/site-packages/airflow/models/dagbag.py", line 351, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/airbyte/airflow/dags/cosmos_profile_mapping_dag.py", line 41, in <module>
    cosmos_dag_using_profile_mapping_dag()
  File "/home/airbyte/devtools/cosmos/cosmos-venv/lib/python3.10/site-packages/airflow/models/dag.py", line 4081, in factory
    f(**f_kwargs)
  File "/home/airbyte/airflow/dags/cosmos_profile_mapping_dag.py", line 32, in cosmos_dag_using_profile_mapping_dag
    transform_data = DbtTaskGroup(
  File "/home/airbyte/devtools/cosmos/cosmos-venv/lib/python3.10/site-packages/cosmos/airflow/task_group.py", line 28, in __init__
    DbtToAirflowConverter.__init__(self, *args, **specific_kwargs(**kwargs))
  File "/home/airbyte/devtools/cosmos/cosmos-venv/lib/python3.10/site-packages/cosmos/converter.py", line 247, in __init__
    self.dbt_graph.load(method=render_config.load_method, execution_mode=execution_config.execution_mode)
  File "/home/airbyte/devtools/cosmos/cosmos-venv/lib/python3.10/site-packages/cosmos/dbt/graph.py", line 325, in load
    self.load_via_dbt_ls()
  File "/home/airbyte/devtools/cosmos/cosmos-venv/lib/python3.10/site-packages/cosmos/dbt/graph.py", line 367, in load_via_dbt_ls
    self.load_via_dbt_ls_without_cache()
  File "/home/airbyte/devtools/cosmos/cosmos-venv/lib/python3.10/site-packages/cosmos/dbt/graph.py", line 475, in load_via_dbt_ls_without_cache
    nodes = self.run_dbt_ls(dbt_cmd, self.project_path, tmpdir_path, env)
  File "/home/airbyte/devtools/cosmos/cosmos-venv/lib/python3.10/site-packages/cosmos/dbt/graph.py", line 348, in run_dbt_ls
    stdout = run_command(ls_command, tmp_dir, env_vars)
  File "/home/airbyte/devtools/cosmos/cosmos-venv/lib/python3.10/site-packages/cosmos/dbt/graph.py", line 120, in run_command
    raise CosmosLoadDbtException(f"Unable to run {command} due to the error:\n{details}")
cosmos.dbt.graph.CosmosLoadDbtException: Unable to run ['/home/airbyte/devtools/cosmos/cosmos-venv/bin/dbt', 'ls', '--output', 'json', '--project-dir', '/tmp/tmp84a8an_e', '--profiles-dir', '/tmp/cosmos/profile/cf3dc9700b875516f605be1e8410b5dc7d9765fe25b7da42140dfc441de87144', '--profile', 'generated_profile', '--target', 'dev'] due to the error:
06:48:04  Running with dbt=1.8.3
06:48:05  Encountered an error:
Runtime Error
  Credentials in profile "generated_profile", target "dev" invalid: Runtime Error
    Must specify `schema` in profile

How to reproduce

  1. create venv using python3 -m venv cosmos-env
  2. Install 1.6.0a1 - pip install astronomer-cosmos==1.6.0a1
  3. Install dbt-teradata and dbt-core - pip install dbt-teradata dbt-core
  4. Define valid DAG by utilizing TeradataUserPasswordProfileMapping and move to dags directory under airflow home directory
  5. start airflow standalone instance using airflow standalone

Anything else :)?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Contact Details

[email protected]

@sc250072 sc250072 added bug Something isn't working triage-needed Items need to be reviewed / assigned to milestone labels Jul 10, 2024
@dosubot dosubot bot added area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc area:profile Related to ProfileConfig, like Athena, BigQuery, Clickhouse, Spark, Trino, etc dbt:list Primarily related to dbt list command or functionality execution:virtualenv Related to Virtualenv execution environment parsing:dbt_ls Issues, questions, or features related to dbt_ls parsing labels Jul 10, 2024
Copy link

dosubot bot commented Jul 10, 2024

Hey @sc250072, I'm here to help you with the issue you're facing. I can assist with bug fixes, answer questions, and guide you on how to contribute. Let's tackle this together! 🤖

The error "Must specify schema in profile" occurs because the schema field is a required field in the profile mapping for Teradata in Astronomer Cosmos version 1.6.0a1. The schema field is mapped to the database field in the Airflow connection, and it must be specified in your profile configuration.

To resolve this error, ensure that your mock Teradata profile includes the schema field. Here is an example of how to include the schema field in your profile configuration:

from cosmos.config import ProfileConfig
from cosmos.profiles import TeradataUserPasswordProfileMapping  # Assuming TeradataUserPasswordProfileMapping exists

profile_config = ProfileConfig(
    profile_name="default",
    target_name="dev",
    profile_mapping=TeradataUserPasswordProfileMapping(
        conn_id="my_teradata_conn_id",
        profile_args={"schema": "my_schema"},
    ),
)

dag = DbtDag(profile_config=profile_config, ...)

Additionally, ensure that your test cases include the schema field in the task_args:

def test_validate_arguments_schema_in_task_args():
    profile_config = ProfileConfig(
        profile_name="test",
        target_name="test",
        profile_mapping=TeradataUserPasswordProfileMapping(conn_id="test", profile_args={}),
    )
    task_args = {"schema": "abcd"}
    validate_arguments(
        select=[], exclude=[], profile_config=profile_config, task_args=task_args, execution_mode=ExecutionMode.LOCAL
    )
    assert profile_config.profile_mapping.profile_args["schema"] == "abcd"

Make sure your profile configuration includes the schema field in a similar manner to avoid the error [1][2][3].

To continue talking to Dosu, mention @dosu.

@tatiana tatiana added this to the Cosmos 1.6.0 milestone Jul 10, 2024
arojasb3 pushed a commit to arojasb3/astronomer-cosmos that referenced this issue Jul 14, 2024
…ofile", target "dev" invalid: Runtime Error Must specify `schema` in Teradata profile (astronomer#1088)

`TeradataUserPassword` profile mapping throws below error for mock
profile

```
Credentials in profile "generated_profile", target "dev" invalid:
Runtime Error Must specify the schema in Teradata profile
```

Closes astronomer#1087
@tatiana tatiana mentioned this issue Aug 16, 2024
18 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc area:profile Related to ProfileConfig, like Athena, BigQuery, Clickhouse, Spark, Trino, etc bug Something isn't working dbt:list Primarily related to dbt list command or functionality execution:virtualenv Related to Virtualenv execution environment parsing:dbt_ls Issues, questions, or features related to dbt_ls parsing triage-needed Items need to be reviewed / assigned to milestone
Projects
None yet
2 participants