Databricks SQL Analytics endpoint does not support set statements

[jtcohen6]

Describe the bug

We use set statements as part of the incremental materialization, specifically for the insert_overwrite strategy with partitioned tables:

https://github.com/fishtown-analytics/dbt-spark/blob/0e75069c3c223546ecec2c6038534d826e9a7dd8/dbt/include/spark/macros/materializations/incremental.sql#L103-L111

This is currently causing a failure in our integration tests.

Incidentally, the SQL Analytics endpoint doesn't support the merge strategy either, since it requires creating a temporary view as currently written. Databricks has promised that create temp view will eventually be supported by the new endpoint.

Steps To Reproduce

Try running any incremental model while connected to a SQL Analytics endpoint

Expected behavior

The weird thing is that set statements used to work. I only saw this crop up in the integration tests for #132; something has definitely changed in the last few weeks.

[Fokko]

Weird stuff. Did the Databricks Runtime change?

It works with Runtime 7.5:

[jtcohen6]

@Fokko That's just the problem: this works fine on interactive / all-purpose clusters running the latest Databricks runtime versions; it's not working on SQL Analytics endpoints (accessible via the new / preview Redash query editor).