RDB Shredder: persist synthetic duplicates on disk

[chuwy]

Currently, RDB Shredder job has a lineage with three RDDs branched off from goodWithSyntheticDupes, an RDD where non-determenistic UUID.randomUUID() operation invoked. It means that if Spark has not enough memory for storing a block with synthetic duplicates - it can evict it and recompute afterwards for one of derived RDDs. During that recomputation a new event_id will be generated that won't correspond to events, which leads to a scenario with orphaned shredded entities.

In other words, it leads to a scenario where single enriched event is shredded into three entities:

1. events - atomic TSV
2. jsons - all shredded entities: contexts and self-describing events (except iglu:com.snowplowanalytics.snowplow/duplicate/jsonschema/1-0-0 context)
3. duplicate context

won't have same event_id/root_id, which means they could not be joined in Redshift.

Orphans appear only for events went though synthetic deduplication, i.e. events that have same event_id, but different fingerprints. These events usually are generated by bots, not real users and rarely exceed 0.5% of events.

Restructuring RDD lineage (using new Analytics SDK 0.4.0) to union jsons with duplicate and persisting an RDD with newly generated event_ids with MEMORY_AND_DISK_SER storage should solve the problem.