PM-524 - Mina Daemon - Implement Precomputed block upload on s3 #136
Conversation
kaozenn
force-pushed
the
PM-524-implement-precomputed-block-upload-for-punk-poll
branch
3 times, most recently
from
08b04bc
to
d5f50d5
Compare
kaozenn
force-pushed
the
PM-524-implement-precomputed-block-upload-for-punk-poll
branch
from
d5f50d5
to
9bf2c41
Compare
|
|
||
| # This script is used to impersonate gcloud in the mina-daemon container and store the downloaded blocks in the /blocks directory | ||
|
|
||
| echo "gcloud authentication disabled by gcloud-impersonator.sh" |
There was a problem hiding this comment.
is this a placeholder for implementation later?
There was a problem hiding this comment.
No, It is just because gcloud binary is required, for the upload-to-gcloud flag. So I need to have also a fake script that can be executed to bypass the gcloud authentication
| if aws s3 ls "s3://${S3_BUCKET}/${block_name}" 2>&1; then | ||
| echo "=> File ${block_name} already exists in S3. Skipping." | ||
| else |
There was a problem hiding this comment.
I can't believe that aws s3 does not have a flag like gsutil have -n. Because essentially you're making 2 api calls per file
There was a problem hiding this comment.
Yes, I agree. I'll keep that PR open and will double check it
There was a problem hiding this comment.
I checked it, there seem to be nothing on s3 api. Also sync api seems to be useless as with many objects it will become very slow very quick.
However I saw there's a flag --if-none-match on s3api api
Also I see this very old thread that has been opened/closed/opened for years
aws/aws-cli#2874
| {{- if .Values.deployment.storeBlocks.aws.enabled }} | ||
| - name: {{ .Release.Name }}-scripts | ||
| mountPath: /usr/bin/gsutil | ||
| subPath: gsutil-impersonator.sh |
There was a problem hiding this comment.
I like that you kept a clear separation of what is a script and what it impersonates!
kaozenn
deleted the
PM-524-implement-precomputed-block-upload-for-punk-poll
branch
Sorry about this bulky PR, it was not easy to refactor the gcp logic and incorporate the s3 one