-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add --no-overwrite option to aws s3 cp/mv #2874
Comments
Marking as a feature request. The tricky part if we did it in |
I'd like to see this in The reason I don't use When you run When you run On a sufficiently large destination bucket, Obviously fixing |
I'm also very interested in this feature. An optional interactive prompt for overwriting files would also be nice to have. |
yes @sgrimm-sg, it makes sense. I am also interested to see CLI cp command which can actually handle these conditions. |
It would be extremely useful for this to be an option on |
Good Morning! We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI. This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports. As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions. We’ve imported existing feature requests from GitHub - Search for this issue there! And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue. GitHub will remain the channel for reporting bugs. Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface -The AWS SDKs & Tools Team This entry can specifically be found on UserVoice at: https://aws.uservoice.com/forums/598381-aws-command-line-interface/suggestions/33168406-add-no-overwrite-option-to-aws-s3-cp-mv |
Related:
|
Based on community feedback, we have decided to return feature requests to GitHub issues. |
@jamesls that's great! can you please respond to the suggest at hand? --no-overwrite would be a great addition and it will avoid wrapping the calls with scripts |
+1 to this issue. I propose -n, --no-clobber to match existing Linux cp command options. |
Has there been any implementation of this request? Trying to work with Windows batch files to do local backup > S3, this is the easiest methods, a simple no-overwrite or similar flag. |
Any update regarding this feature? |
Any update regarding this feature ? Thanks |
Any update regarding this feature? |
1 similar comment
Any update regarding this feature? |
sev3, +1 |
Really need this feature added as S3 sync does not seem to upload every file. |
Any updates or workarounds? |
I had to write a python script to load all of the items in the bucket into an array (list), then load all the items from the directory I want to sync, then compare the arrays and upload the local items not in the S3 array. |
@southpaw5271 - care to share your script and save me some time ? ; ) |
I don't seem to have it anymore :( Sorry! |
This flag would also be valuable for the
|
We also need the --no-overwrite option from s3 to local. We've been burned by accidental overwrites from well-meaning individuals, and this would be a very much appreciated way to put up a "guardrail" for them. Thanks! |
any update? |
@jfstephe, unfortunately not. I'll keep checking! |
@kdaily Any news on this MR? Thanks |
FYI I developed this with a colleague. It's an S3-only based way of ensuing that only one person will be updating an S3 file at any one time. It's javascript based. Not suggesting this is better than AWS supporting the requested feature, and not suggesting this is better than using a database, but for us this was the best option for now, and may prove useful for others. |
Really a needed feature. Still waiting for a solution to this. |
Another year. Object storage doesn't support Mutex, Browser doesn't support PWA, WebAssembly no 2.0, you can convince now they are zenophobic |
Is this really something that cannot be developed in under 5 years? |
it's been 5 years, and no progress |
Not to bother people. But I'll provide a use case for this. There was a timing bug in our build service. Which ended making our servers build the latest code into packages thinking it was the previous version. The worst part is that was a release build. That glitch caused an impostor version to be uploaded to our S3 storage and people downloaded it. A Please add the flag. |
It costs amazon money to support this because no-overwrite means there is going to be a single thread bottle neck which will be looking at whether the object exist Essentially, the request itself requires the server to compute whether something exists, without requests from other origin causing racing problem But they should eat this cost, instead of having us routing all the request first to our own server / worker to implement this check manually, for once the company should offer this tiny feature S3 is object storage, their selling point is infinite scalability, amazon's refusal on offer this would show a rather poor posture from the tech giant Meanwhile, amazon's pricing really says a lot about making things cheaper and fair for both player, although other platforms might be catching up with all the unlimited request and egress features too |
This should be clearly stated in the docs. I think most of the time this is not an issue (at least for a good amount of the usecases).
I would happily eat the cost myself. The dev time spent on implementing this manually is 1000 times more than I'll ever spend on the request costs. |
absolutely pitch-perfect AWS |
Has anyone come up with a x-ish one-liner to overcome this? |
See comment above re: https://github.com/jfstephe/aws-s3-lock . May help? |
This can be done with the |
That's a great idea. This seems like it would be trivial to implement with a flag if that actually works... |
https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/ , https://docs.aws.amazon.com/AmazonS3/latest/userguide/conditional-requests.html#conditional-writes |
Is there a plan to implement <Error>
<Code>NotImplemented</Code>
<Message>A header you provided implies functionality that is not implemented</Message>
<Header>If-None-Match</Header
<RequestId>REDACTED</RequestId
<HostId>REDACTED</HostId>
</Error> |
Okay, so while
|
It would be nice to have a convenience function
--no-overwrite
foraws s3 cp/mv
commands, which would check the target destination doesn't already exist before putting a file into an s3 bucket.Of course this logic couldn't be guaranteed by the AWS API (afaik...) and is vulnerable to race conditions, etc. But it would be helpful to prevent unintentional mistakes!
The text was updated successfully, but these errors were encountered: