Allow DBT Deps Artifact URL To Be Configurable

[DustinMoriarty]

Many companies have build systems which do not allow network traffic outside of the private network and instead rely on mirrors in private artifact repositories. Other package managers such as npm, pip and maven allow the source URL to be configured to point to a private mirror. This allows all of the same version resolution features to work and allows sub-dependencies to be resolved. However, DBT does not have a mature solution for enabling private mirrors. Once the solution is implemented by DBT, it will take some time for private repository providers (e.g. JFrog) to implement the standard on their side. However, they have done this for all other languages, so there is no reason to think they would not for DBT provided that the standard is reasonably easy to follow and well documented.

Alternatives

1. We have investigated using the git option in DBT deps. This option requires that we mirror git repositories. Git is not the normal place for mirroring so it is out of process. Git is a version control system, not an artifact repository. Build systems assume that mirroring occurs in the artifact management system. In addition, this does not resolve the problem of child dependencies which point to the public DBT repo.
2. We have investigated using the tar option. This option does work with artifact repositories which do support paths to tar files. However, it does not support version resolution or child dependencies which point to a public repo.
3. We are currently looking at vendoring all of our dependencies in the repo that uses the packages. This of course is a very gross solution which defeats the whole point of package management. It inflates the size of the repo and relies on local management of packages to be done correctly prior to commit.

Examples of Dependency Management in Mature Package Management Systems

1. In pip, the INDEX_URL and the EXTRA_INDEX_URL can be configured with an environment variable. The URL contains the basic auth credentials. Tokens use basic auth.
2. In python poetry, the source repositories can be specified in the pyproject.toml file and tied to certain packages. The credentials are specified in a separate config file. The pyproject.toml file is checked into version control and the global config file is stored separately and not checked in. The config file and the pyproject.toml files are managed by the CLI.
3. In maven, the repository URL can be specified in the pom.xml file with credentials specified in the settings.xml file. The pom.xml file is checked into version control and the settings.xml file is stored in a different location and not checked in.
4. In npm, the repository URL can be specified in a config file and set via CLI.

Requirements

1. Configurable Package URL. Ideally, one of these configuration methods should use an environment variable since most CI/CD systems use environment variables as the method to manage configurable data and secrets.
2. Ability to handle authorization and secrets without committing the secret to version control. The secret can be in the URL using basic auth. However, that means that the URL cannot be in the packages.yml file unless DBT stores the basic auth credentials separately and injects them into the URL. DBT does not need to figure out the actual auth solution, just how the secrets are handled on the CLI side.
3. Well documented easy to follow standard for the package index itself. I am not sure how the DBT package system works.

[DustinMoriarty]

It looks like maybe there is an environment variable for this. However, I don't see any documentation on it. Perhaps I could help to document this if someone can help to provide some insight about if this feature is intended to be supported long term and if there are some other technical details that I should be aware of. It looks to me like one could just provide the environment variable pointing to a different base URL and then host the index with all the tar file locations changed to local ones. Am I wrong?

https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/clients/registry.py#L21