Common: use run manifest

[chuwy]

This looks like a most bullet-proof solution against inconsistent S3. Instead of relying on listing S3, we can collect data in shredder (like we do for Snowflake) and write to external manifest (DynamoDB).

Unlike consistency check this does not add idle time and should be very reliable.

Should be optional to reduce maintainance routine for pipelines that don't suffer from inconsistency.

[alexanderdean]

Let's just go for this one. It will solve the problem and be much more robust than our consistency check...

[chuwy]

See snowplow/snowplow-scala-analytics-sdk#45

[chuwy]

Just found another quick and dirty solution to consistency problem (not going to implement it as it is dangerous, just sharing).

For example, last ETL had run=2018-01-01-12-00-00 id and left few ghost folders. Current run is run=2018-01-01-18-30-00 and fails because of files from previous run. We can simply discard first folder because assumption that older folder is always a ghost is relatively safe. It is interesting that StorageLoader always tried to load only first directory. If it was always last directory - we wouldn't even find this problem probably.

[alexanderdean]

Makes sense @chuwy, thanks for sharing.