DBT errors out when using BigQuery datasets that begin with underscores

[ghost]

In BigQuery, datasets that begin with underscores are hidden in the UI and API. As a result, when DBT checks for the existence of a dataset, it doesn't find it, tries to create it, then errors out because the dataset already exists. There are a couple of ways to handle this:

1. Prevent it by checking for the existence of the dataset by a different method, or
2. Handle the exception being thrown by BigQuery.

Prevent the Exception

I think this can be fixed by adding include_all=True as a parameter in list_datasets() and changing the create_schema method to check for the existence of the dataset id in the output of get_existing_schemas instead of using get_dataset

Handle the Exception

When this occurs, BigQuery responds with a 409 response code: 409 POST https://www.googleapis.com/bigquery/v2/projects/<project_id>/datasets: Already Exists: Dataset <project_id>:<table_name>
create_schema could be modified to interpret a 409 response code as success.

[drewbanin]

Thanks @rsmichaeldunn! This feature is news to me. Should be pretty straightforward to pass the include_all param in here

[ghost]

It was news to me too. I was wondering where on earth my missing dataset was. I haven't found it documented by Google anywhere, and there's only one stackoverflow article on it that I found: https://stackoverflow.com/questions/19524516/dataset-with-name-starting-with-an-underscore-doesnt-show-up-in-google-bigquery

[drewbanin]

Fixed in #954