Feature request: Add a dbt flag to specify the working directory

[heisencoder]

Feature

Feature description

Add a new flag to dbt to allow specifying the directory of the dbt_project.yml file (which in turn defines the relative directories of the other project files). This would be analogous to the -C and --git-dir flags for the git command. This would also complement the existing --profiles-dir flag.

Suggested name: --working-dir

Who will this benefit?

It would make tooling easier if it wasn't necessary to change the working directory before running dbt. For example, this reference implementation that combines dbt with airflow needs to include a 'cd' command within each BashOperator: https://gist.github.com/adamhaney/916a21b0adcabef4038c38e3ac96a36f
With this change, it would be sufficient just to pass this flag each time.

Also see slack discussion: https://getdbt.slack.com/messages/CBSQTAPLG/convo/CBSQTAPLG-1560546908.162800/

[heisencoder]

I took a quick crack at this implementation, and it's a pretty straightforward modification to main.py in the latest release (0.11.1). However, this change runs into major merge conflicts with 1a0df17?diff=split when merged to dev/wilt-chamberlain, so I'm going to abandon this change for now (until wilt-chamberlain is released).

Also, I think a better flag name is --project-dir instead of --working-dir, since this is the directory for the project.

[beckjake]

Do you mean 0.13.1 when you say latest release?

I am not really surprised by that, that was a heck of a merge.

Fortunately, I think the place to do this is 1) off wilt-chamberlain (or louisa-may-alcott once I merge in the current state of wilt-chamberlain) 2) in the run_from_args function, so maybe not too bad? Just look at parsed and issue an os.chdir into parsed.project_dir if it's specified, I assume...

[heisencoder]

Yup! I meant 0.13.1. I'll take a quick look at run_from_args to see if that's easy in wilt-chamberlain.

[beckjake]

Ok. it's possible I have overlooked something and the real place to do it is handle_and_check (the caller of run_from_args), right after parse_args, but I don't think anything there cares about your current working directory.

[heisencoder]

The change I was planning against 0.13.1's main.py looked basically like this:

# (Define new flag named 'project-dir' and make it optional.)

def get_nearest_project_dir(parsed):
    # If the user provides an explicit project directory, use that.
    if parsed.project_dir:
        project_file = os.path.join(parsed.project_dir, "dbt_project.yml")
        if os.path.exists(project_file):
            return parsed.project_dir
        else:
            return None

    # Otherwise, use the current working directory and all its parents.
    cwd = os.getcwd()

    root_path = os.path.abspath(os.sep)
    while cwd != root_path:
        project_file = os.path.join(cwd, "dbt_project.yml")
        if os.path.exists(project_file):
            return cwd
        cwd = os.path.dirname(cwd)

    return None

So, wherever the 'get_nearest_project_dir' moved to, that's where I was thinking it would go. I didn't see anything in the new main.py that did an os.chdir(nearest_project_dir), so I'm assuming that got moved into a different file (or is now done via a different mechanism).

[beckjake]

Oh, I see! That's reasonable enough, I hadn't thought of that. It should now be in core/task/base.py, you'll probably have to change move_to_nearest_project_dir and it to take arguments again, which is fine! The interface I think we care about preserving is BaseTask.from_args, anything under that is just implementation details.

In that method, args is a parsed argparse.Namespace, so all the same ideas should apply.

[heisencoder]

Good to know! That should make it a straightforward change then.

[heisencoder]

Hi @beckjake, where should I put the unit test for this change?

[beckjake]

There's a set of tests in test/unit/test_config.py that deal with this sort of thing, in particular TestProjectFile and TestRuntimeConfigFiles. You'll have to update the Args object defined in there but I kind of assume you'll have to do that anyway.

[heisencoder]

Hi @beckjake: Let me know if the unit test looks good. I couldn't get TestProjectFile and TestRuntimeConfigFiles to easily invoke the modified code, so I added a new test case called TestRunOperationTask to create a new RunOperationTask and make sure the os directory was appropriately changed. If the test looks good, this change is ready for merging.

[drewbanin]

closed by #1549