Bug Hitting ThrottlingException on GetWorkGroup with threads turned up

[nicor88]

Discussed in #591

^{Originally posted by dacreify March 6, 2024}
Recently we started hitting ThrottlingException on GetWorkGroup calls at this spot in the code:

https://github.com/dbt-athena/dbt-athena/blob/a1b8db5de90b20557bcd5e0c51a30177bcddaa5f/dbt/adapters/athena/impl.py#L231

From CloudTrail it looks like dbt-athena winds up making a GetWorkGroup call for every model run. There's no documented quota for this call but obviously we're hitting one. Actual StartQueryExecution can go 20/second or burst to 80/second so GetWorkGroup definitely seems to be below that in any case.

Anyone else hit this? Can we cache the results of GetWorkGroup to avoid it?

[nicor88]

@dacreify @juliansteger-sc could you try the latest version from the main branch and verify that the fix solves your issue?

[dacreify]

@nicor88 I installed from main and cranked the threads back up on our project. I'm not able to reproduce the throttling now, but I guess it's hard to tell if this is the dbt-athena caching or AWS fixing their regression.

I noticed that the cache key includes the client object:

https://github.com/dbt-athena/dbt-athena/blob/3cdd7ee96f39b347179f729295d8d5c69f140d69/dbt/adapters/athena/impl.py#L219-L224

Given the module-level lock here I'm guessing there's only one instance of athena_client in use for the whole run?

https://github.com/dbt-athena/dbt-athena/blob/3cdd7ee96f39b347179f729295d8d5c69f140d69/dbt/adapters/athena/impl.py#L232-L240

Just confirming that I understand how the caching is working.

[nicor88]

@dacreify lru_cache allow you to cache result in case the same inputs are passed multiple times.
Given the fact that client and workgroup don't cange it should work as expected.
Not sure when using more threads if multiple clients are spawn per thread, I needed to check this behavior, in general lru_cache is thread safe.

[juliansteger-sc]

I'm not able to reproduce the throttling now, but I guess it's hard to tell if this is the dbt-athena caching or AWS fixing their regression.

same for us, but fix looks reasonable. thanks for investigating & fixing

[nicor88]

Let's close this issue for now, given the fact that either AWS or the caching behaviour helped to mitigate the issue.