Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RDB Loader: fix eventual consistency problem #3113

Closed
alexanderdean opened this issue Feb 14, 2017 · 1 comment
Closed

RDB Loader: fix eventual consistency problem #3113

alexanderdean opened this issue Feb 14, 2017 · 1 comment
Assignees
@chuwy
Labels
4. Storage
Milestone
R90 Lascaux

Comments

@alexanderdean
Copy link
Member

We have a problem with users who a) have large volumes and b) have high event/context cardinalities where:

  • The StorageLoader code for discovering folders in S3 of shredded entities detects already-deleted "ghost" folders (this is due to S3's notorious eventual consistency problems), but
  • The Redshift COPY statement (correctly) does not find this ghost folder (nor any contents of it), and thus
  • The overall COPY statement (correctly) fails

We need to fix this - we can hope that just porting the StorageLoader S3 folder discovery code might make the problem go away, but it would be better to actively guard against this issue in this release.

/cc @leonmaas
@alexanderdean alexanderdean added this to the R9x [HAD] StorageLoader reboot milestone Feb 14, 2017
@chuwy chuwy changed the title Relational Database Loader: fix eventual consistency problem RDB Loader: fix eventual consistency problem Feb 20, 2017
@gregbonnette
Copy link

Good talk on this issue and some potential solutions... https://spark-summit.org/east-2017/events/spark-and-object-stores-what-you-need-to-know/

chuwy added a commit that referenced this issue Jun 27, 2017
@chuwy
RDB Loader: fix eventual consistency problem (close #3113)
ab88075
chuwy added a commit that referenced this issue Jun 28, 2017
@chuwy
RDB Loader: fix eventual consistency problem (close #3113)
272ebe0
chuwy added a commit that referenced this issue Jun 29, 2017
@chuwy
RDB Loader: fix eventual consistency problem (close #3113)
a7dce72
chuwy added a commit that referenced this issue Jun 30, 2017
@chuwy
RDB Loader: fix eventual consistency problem (close #3113)
0c53fb8
chuwy added a commit that referenced this issue Jul 3, 2017
@chuwy
RDB Loader: fix eventual consistency problem (close #3113)
b9808d1
chuwy added a commit that referenced this issue Jul 8, 2017
@chuwy
RDB Loader: fix eventual consistency problem (close #3113)
26d5fb8
chuwy added a commit that referenced this issue Jul 26, 2017
@chuwy
RDB Loader: fix eventual consistency problem (close #3113)
5efce0e
@chuwy chuwy closed this as completed in a9e0a08 Jul 26, 2017
BenFradet pushed a commit that referenced this issue Aug 2, 2017
@chuwy @BenFradet
RDB Loader: fix eventual consistency problem (close #3113)
defe0bb
This was referenced Aug 21, 2017
Closed
Closed
chuwy added a commit to snowplow/snowplow-rdb-loader that referenced this issue Sep 5, 2017
@chuwy
RDB Loader: fix eventual consistency problem (snowplow/snowplow#3113)
2dccaad
@chuwy chuwy mentioned this issue Nov 3, 2017
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chuwy chuwy

Labels
4. Storage
Projects
None yet
Milestone
R90 Lascaux
Development

No branches or pull requests
3 participants
@alexanderdean @chuwy @gregbonnette