[CT-473] [Bug] dbt deps still can fail with 'NoneType' object is not subscriptable

[barberscott]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

It appears that there is still an exception that the current code is not catching on v1.0.5-rc2 -- this one is new!

Traceback (most recent call last):
  File "/usr/src/app/sinter/clients/dbt.py", line 1333, in call
    dbt_main.handle(command + extra_args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 152, in handle
    res, success = handle_and_check(args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 192, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 246, in run_from_args
    results = task.run()
  File "/usr/local/lib/python3.8/dist-packages/dbt/task/deps.py", line 56, in run
    final_deps = resolve_packages(packages, self.config)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/resolver.py", line 135, in resolve_packages
    target = final[package].resolved().fetch_metadata(config, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/registry.py", line 150, in resolved
    available = registry.get_available_versions(self.package)
  File "/usr/local/lib/python3.8/dist-packages/dbt/clients/registry.py", line 131, in get_available_versions
    response = package(package_name)
  File "/usr/local/lib/python3.8/dist-packages/dbt/clients/registry.py", line 120, in package
    return response["versions"]
TypeError: 'NoneType' object is not subscriptable

so something else is sneaking through.

Expected Behavior

The root cause of this error is caught, logged, and the download retried.

Steps To Reproduce

No response

Relevant log output

No response

Environment

- dbt: 1.0.5-rc2 (dbt Cloud)

What database are you using dbt with?

No response

Additional Context

No response

[jtcohen6]

Thanks for the report @barberscott!

I'm trying to wrap my head around how, if response is None, it still managed to clear all of these checks, rather than encountering one of these many other exceptions:

dbt-core/core/dbt/clients/registry.py

Lines 54 to 90 in 5852f17

	if not isinstance(response, dict): # This will also catch Nonetype
	error_msg = (
	f"Request error: Expected a response type of <dict> but got {type(response)} instead"
	)
	fire_event(RegistryResponseUnexpectedType(response=response))
	raise requests.exceptions.ContentDecodingError(error_msg, response=resp)
	# check for expected top level keys
	expected_keys = {"name", "versions"}
	if not expected_keys.issubset(response):
	error_msg = (
	f"Request error: Expected the response to contain keys {expected_keys} "
	f"but is missing {expected_keys.difference(set(response))}"
	)
	fire_event(RegistryResponseMissingTopKeys(response=response))
	raise requests.exceptions.ContentDecodingError(error_msg, response=resp)
	# check for the keys we need nested under each version
	expected_version_keys = {"name", "packages", "downloads"}
	all_keys = set().union(*(response["versions"][d] for d in response["versions"]))
	if not expected_version_keys.issubset(all_keys):
	error_msg = (
	"Request error: Expected the response for the version to contain keys "
	f"{expected_version_keys} but is missing {expected_version_keys.difference(all_keys)}"
	)
	fire_event(RegistryResponseMissingNestedKeys(response=response))
	raise requests.exceptions.ContentDecodingError(error_msg, response=resp)
	# all version responses should contain identical keys.
	has_extra_keys = set().difference(*(response["versions"][d] for d in response["versions"]))
	if has_extra_keys:
	error_msg = (
	"Request error: Keys for all versions do not match. Found extra key(s) "
	f"of {has_extra_keys}."
	)
	fire_event(RegistryResponseExtraNestedKeys(response=response))
	raise requests.exceptions.ContentDecodingError(error_msg, response=resp)

Ideally, that exception would trigger a retry; at the very least, it should be the exception we see in the stacktrace, rather than the surprise at trying to resolve response["versions"].

Accessing the raw logs may give a better sense of the narrative, for exactly which steps succeeded in the leadup to the package call at the end of the stacktrace. The only thing I can think is that we may need to switch the order of retry and caching, so that _get_with_retries -> _get_cached rather than _get_cached -> _get_with_retries.

[gshank]

I poked through the code too and didn't see how that could happen. At the very least, we should check for if "versions" in response: ... else: <dump information and throw error>

[gshank]

I do wonder if we should catch all exceptions in connection_exception_retry. We're only catching RequestException and ReadError. Should we also add a catch of Exception? Not catching an exception there shouldn't cause this error as far as I can tell, but just to make sure there's not something we're missing...

[barberscott]

Reopening since it looks like, darn, we didn't catch this!

A recent dbt run failed on v1.0.5 with the following error:

Traceback (most recent call last):
  File "/usr/src/app/sinter/clients/dbt.py", line 1338, in call
    dbt_main.handle(command + extra_args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 152, in handle
    res, success = handle_and_check(args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 192, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 246, in run_from_args
    results = task.run()
  File "/usr/local/lib/python3.8/dist-packages/dbt/task/deps.py", line 56, in run
    final_deps = resolve_packages(packages, self.config)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/resolver.py", line 135, in resolve_packages
    target = final[package].resolved().fetch_metadata(config, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/registry.py", line 150, in resolved
    available = registry.get_available_versions(self.package)
  File "/usr/local/lib/python3.8/dist-packages/dbt/clients/registry.py", line 131, in get_available_versions
    response = package(package_name)
  File "/usr/local/lib/python3.8/dist-packages/dbt/clients/registry.py", line 120, in package
    return response["versions"]
TypeError: 'NoneType' object is not subscriptable