[CT-528] [Bug] Local package relative paths in packages.yml are converted to absolute rather than relative symlinks

[mbhynes]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Problem

• dbt allows local packages to be specified in the packages.yml, such as:

packages:
  - local: ../common

• These kind of relative package paths are a convenient and straightforward way of managing dependencies in a large organization with project structures such as the following, in which some local projects are "common" or shared macros across a company, e.g. in a common project:

org_monorepo/
- org_monorepo/dbt/
  - org_monorepo/dbt/project1/
  - org_monorepo/dbt/project2/
  - org_monorepo/dbt/common

• However the relative paths specified in the packages.yml are converted to absolute paths without the user's direction in the core/dbt/clients/system.resolve_path_from_base function:

• This is unexpected and undesirable behaviour, especially when the relative tree structure is desired in source code management & deployment; e.g. the devs will write on their local machines, but CI/CD and deployment occurs on other machines in which the absolute path of the symlink doesn't exist.

Expected Behavior

If a relative path is given, then the symlink should be this relative path.

After all--- user has specified relative for a reason, and could provide an absolute path themselves if that was the desired behaviour.

Steps To Reproduce

Please see the example above; any relative path provided in a packages.yml gets converted to an absolute symlink, when it should be converted to a relative symlink

Relevant log output

No response

Environment

- OS: Mac 11.6
- Python: (multiple), but Python 3.9
- dbt:

installed version: 0.20.0
Plugins:
  - bigquery: 0.20.0

### What database are you using dbt with?

_No response_

### Additional Context

_No response_

[gshank]

Thanks for reporting this. I can see that it would be much more convenient for your use case to have relative symlinks. The relevant code is in core/dbt/deps/local.py, LocalPinnedPackage.install. It seems like we could check for the existence of the absolute path, but symlink the relative path.

I'm not sure if there would be operating system complications with this though. Windows can sometimes be an issue. We'll look into this.

[nathaniel-may]

@gshank, my understanding is that the issue would arise if you run dbt deps on your local machine then copy the dependencies somewhere with a different file structure (like a different machine for deployment) and attempt to run dbt run from there. Is this the only problem this would cause? Is this a method we support for deploying a dbt project?

CC: @jtcohen6

[Raalsky]

Hey, I'm not sure how it's correlated with the above - let me know if I should move to a new issue 😉.

I have following structure:

projects:
  - utils
  - mailing
    - integration_tests

Where utils, mailing and integration_tests are dbt projects. Placing of integration_tests were suggested in dbt docs.

Now mailing project has following package.yml:

packages:
  - local: ../utils

And integration_tests - package.yml:

packages:
  - local: ../

Now running dbt deps inside of integration_tests failures with:

21:04:28  Running with dbt=1.1.0
21:04:29  Encountered an error:
Runtime Error
  no dbt_project.yml found at expected path [...]/projects/mailing/utils/dbt_project.yml

This suggests that integration_tests has following "computed" dependencies:

packages:
  - local: ../
  - local: ../utils

Instead of expected one:

packages:
  - local: ../
  - local: ../../utils

It seems that path resolving for utils was made in respect of absolute path of source project instead of dependencies' project path. I'm not sure how much it's lack of knowledge and misexpectations on my side versus real problem.

EDIT: Temporal solutions is placing integration_tests in the same level of utils and mailing but it's not as expressive as could be.

[leahwicz]

Notes: we would need to check the security implications of adding this functionality and being sure we implement it with the proper guardrails in place

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.