RDB Loader can hang for many hours

[alexanderdean]

If it's helpful - the hanging seems to occur after the load transaction.

[chuwy]

How do we know it happened after the load transaction? Otherwise I would think about consequence of snowplow/snowplow#3113.

Was there anything particular about dataset size, shredded type cardinality, etc in that load?

[chuwy]

Following actions happen after load:

1. ANALYZE
2. Send unstruct event to collector (should be non-blocking)
3. Dump logs to S3

I suspect actor system inside Scala Tracker, responsible for 2. we had problem with proper Akka-shutdown before.

[alexanderdean]

How do we know it happened after the load transaction?

I had to log in and check that the data had loaded, to avoid either double-loading or missing the hung load...

I like your theory about #2 too...