Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: sqs based architecture data stack #44

Merged
merged 9 commits into from
Aug 10, 2023

Conversation

vasco-santos
Copy link
Contributor

@vasco-santos vasco-santos commented Jul 31, 2023

Implements SQS based architecture for data stack based on proposed architecture

It also implements the aggregator spec role from storacha/specs#71 relying on filecoin-api and filecoin-client

Needs:

  • fix: upgrade data segment w3up#850 + ignored types issues removed (see ae82fdd)
  • test different group IDs really trigger consumer with only messages from that group id (found docs that say so, but nothing exists in official AWS docs. If ends up being an issue, we need to handle that in the consumers and separate them on the consumer functions)
  • Add DynamoDB stream consumer of piece table to propagate events of piece table insert to the next queue

@vasco-santos vasco-santos force-pushed the feat/sqs-based-architecture-data-stack branch from f02946d to 371f45b Compare July 31, 2023 16:38
@vasco-santos vasco-santos force-pushed the feat/sqs-based-architecture-data-stack branch from 371f45b to 32a42e6 Compare August 1, 2023 08:17
@vasco-santos vasco-santos force-pushed the feat/sqs-based-architecture-data-stack branch from 32a42e6 to f66e4b3 Compare August 4, 2023 08:14
@vasco-santos vasco-santos force-pushed the feat/sqs-based-architecture-data-stack branch from fbe661c to 1004f1e Compare August 7, 2023 10:18
@vasco-santos vasco-santos force-pushed the feat/sqs-based-architecture-data-stack branch from 1004f1e to a794436 Compare August 7, 2023 10:54
@vasco-santos vasco-santos force-pushed the feat/sqs-based-architecture-data-stack branch from a794436 to 55e0028 Compare August 7, 2023 11:51
@vasco-santos vasco-santos force-pushed the feat/sqs-based-architecture-data-stack branch from 55e0028 to 7d55806 Compare August 7, 2023 15:56
@vasco-santos vasco-santos force-pushed the feat/sqs-based-architecture-data-stack branch from 7d55806 to b1ac744 Compare August 7, 2023 19:35
Comment on lines 47 to 97
const pieces = []
for (const b of bufferReferences) {
// eslint-disable-next-line unicorn/prefer-spread
pieces.concat(b.ok?.pieces || [])
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

concat does not mutate any arrays it creates a new one so think this is not going to behave correctly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes good catch, these were the tests I still had in progress. Adding final commit that fixes it

@seed-deploy seed-deploy bot temporarily deployed to pr44 August 9, 2023 10:00 Inactive
@vasco-santos vasco-santos marked this pull request as ready for review August 9, 2023 10:09
@seed-deploy
Copy link

seed-deploy bot commented Aug 9, 2023

View stack outputs

@seed-deploy seed-deploy bot temporarily deployed to pr44 August 9, 2023 18:35 Inactive
@@ -282,7 +282,7 @@ function aggregatePieces (bufferedPieces, sizes) {
return {
addedBufferedPieces,
remainingBufferedPieces,
aggregate: aggregate.toInfo()
aggregate: aggregate
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
aggregate: aggregate
aggregate

@seed-deploy seed-deploy bot temporarily deployed to pr44 August 10, 2023 07:45 Inactive
@vasco-santos vasco-santos merged commit 794f34e into main Aug 10, 2023
1 check passed
@vasco-santos vasco-santos deleted the feat/sqs-based-architecture-data-stack branch August 10, 2023 07:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants