Skip to content
This repository has been archived by the owner on Jul 31, 2020. It is now read-only.

Compaction API #350

Merged
merged 5 commits into from
Oct 28, 2019
Merged

Compaction API #350

merged 5 commits into from
Oct 28, 2019

Conversation

darkdh
Copy link
Member

@darkdh darkdh commented Oct 18, 2019

fix #341
fix #289
brave/brave-browser#6552 will use this API to initiate the compaction

this.withRetry(() => {
return s3Helper.listObjects(this.s3, options, false)
}).then((s3Objects) => {
let latestRecords = {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is going to keep the most recent record in each batch, which is fine, but I think we should add a comment to make that clear

AlexeyBarabash
AlexeyBarabash previously approved these changes Oct 21, 2019
Copy link
Contributor

@AlexeyBarabash AlexeyBarabash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

Copy link
Member

@SergeyZhukovsky SergeyZhukovsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++

@darkdh darkdh force-pushed the compaction branch 3 times, most recently from bd636b1 to 3669788 Compare October 25, 2019 19:25
Copy link
Member

@SergeyZhukovsky SergeyZhukovsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++

compaction for up to 100 records

Add LRU cache(50 records) for missing part for decryption so it can be decrypted in
future round
Copy link
Member

@SergeyZhukovsky SergeyZhukovsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++

@darkdh darkdh merged commit 6f8795a into staging Oct 28, 2019
@darkdh darkdh deleted the compaction branch October 28, 2019 17:52
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement sync record compaction Sync record from SQS is not decoded sometimes
4 participants