"dbt run" can hang

[beckjake]

Issue

dbt run can hang indefinitely

Issue description

If you have a model that another model depends upon, and the materialization of the first model raises an exception that is neither of the type dbt.exceptions.CompilationException nor dbt.exceptions.RuntimeException, dbt run will deadlock.

This issue is reasonably hard to trigger.

Results

dbt dumps a big horrible stack trace and then hangs until I press ctrl+c. I expected it to fail gracefully.

System information

The output of dbt --version:

installed version: 0.12.2

The operating system you're running on:
macOS

The python version you're using (probably the output of python --version)
3.6.5

Steps to reproduce

In as much detail as possible, please provide steps to reproduce the issue. Sample data that triggers the issue, example models, etc are all very helpful here.

1. Create a custom materialization, or just edit the view/table materialization in dbt's internal project. At the first line of the materialization, add {{ exceptions.raise_dependency_error('x') }}.
2. Make a project with a couple models. Make the second one depend upon the first, and configure dbt to use the materialization you edited for the first.

raise_dependency_error is important as it raises an exception that isn't dbt.exceptions.CompilationException or dbt.exceptions.RuntimeException, which are explicitly caught. I'm pretty sure it's possible to trigger exceptions besides those two + this one - we should catch this kind of error and turn it into an error result.

My minimal repro is below:

$ cat macros/oops_materialization.sql
{% materialization oops, default -%}
  {{ exceptions.raise_dependency_error('x') }}
{%- endmaterialization %}
$ cat models/base.sql
{{ config(materialized='oops') }}
select 1 as id
$ cat models/dependent.sql
select * from {{ ref('base') }}
$ cat dbt_project.yml
name: 'debug'
version: '0.1.0'
profile: 'debug'
$ dbt run
Found 2 models, 0 tests, 0 archives, 0 analyses, 96 macros, 2 operations, 0 seed files

14:33:08 | Concurrency: 2 threads (target='default')
14:33:08 |
14:33:08 | 1 of 2 START oops model dbt_postgres.base............................ [RUN]
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/Users/jake/.pyenv/versions/3.6.5/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/Users/jake/.pyenv/versions/3.6.5/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/jake/.pyenv/versions/3.6.5/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/Users/jake/src/fishtown/dbt/dbt/runner.py", line 80, in call_runner
    result = runner.safe_run(self.manifest)
  File "/Users/jake/src/fishtown/dbt/dbt/node_runners.py", line 90, in safe_run
    result = self.run(compiled_node, manifest)
  File "/Users/jake/src/fishtown/dbt/dbt/node_runners.py", line 159, in run
    return self.execute(compiled_node, manifest)
  File "/Users/jake/src/fishtown/dbt/dbt/node_runners.py", line 456, in execute
    materialization_macro.generator(context)()
  File "/Users/jake/src/fishtown/dbt/dbt/clients/jinja.py", line 102, in call
    return macro(*args, **kwargs)
  File "/Users/jake/.pyenv/versions/3.6.5/envs/dbt36old/lib/python3.6/site-packages/jinja2/runtime.py", line 575, in __call__
    return self._invoke(arguments, autoescape)
  File "/Users/jake/.pyenv/versions/3.6.5/envs/dbt36old/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 110, in _invoke
    return original_invoke(self, arguments, autoescape)
  File "/Users/jake/.pyenv/versions/3.6.5/envs/dbt36old/lib/python3.6/site-packages/jinja2/runtime.py", line 579, in _invoke
    rv = self._func(*arguments)
  File "dbt-a42f706d92e89d13921e04ab", line 16, in macro
    to_string(environment.call(context, environment.getattr((undefined(name='exceptions') if l_1_exceptions is missing else l_1_exceptions), 'raise_dependency_error'), 'x')),
  File "/Users/jake/.pyenv/versions/3.6.5/envs/dbt36old/lib/python3.6/site-packages/jinja2/sandbox.py", line 427, in call
    return __context.call(__obj, *args, **kwargs)
  File "/Users/jake/.pyenv/versions/3.6.5/envs/dbt36old/lib/python3.6/site-packages/jinja2/runtime.py", line 262, in call
    return __obj(*args, **kwargs)
  File "/Users/jake/src/fishtown/dbt/dbt/exceptions.py", line 218, in raise_dependency_error
    raise DependencyException(msg)
dbt.exceptions.DependencyException: x

^C
Exited because of keyboard interrupt.

Done. PASS=0 ERROR=0 SKIP=0 TOTAL=0
ctrl-c