dbt fails cryptically on old (I assume?) versions of spark

[beckjake]

With dbt-spark==0.15.3 and dbt==0.15.3, a user in slack reports a couple of errors:

• no support for /* ... */ style comments.
• when query-comment: none is set to avoid that, an error about not enough values to unpack (expected 4, got 1)
  • that's caused by the show table extended ... command only returning one column.

I don't know when spark introduced block comments (2.0 at the latest), but I'm pretty sure show table extended having 4 columns dates to spark 2.2 (around this PR). We should just detect that we're at least above that and bail out if we aren't. I don't think it's unreasonable to require 2.2 as a minimum version, but I'd be very happy to go even higher.

Edit: upon further thought, we should also detect when show table extended ... returns a number of columns that's not 4, and error about it in a better way than tuple-unpacking errors. Alternatively we could access the columns by name, and give a decent error if the column name doesn't exist.

[jtcohen6]

• I'm all for requiring a minimum version of Spark. It looks like 2.2.0 was released in July 2017.
• Do you think we should update this line to check that there are 4 columns before unpacking?

[beckjake]

Yeah, probably to something like:

        for row in results:
            if len(row) != 4:
                raise dbt.exceptions.RuntimeException(
                    f'Invalid value from "show table extended ...", got {len(row)} values, expected 4'
                )
            _schema, name, _, information = row
            rel_type = ('view' if 'Type: VIEW' in information else 'table')