This repository has been archived by the owner on Sep 9, 2020. It is now read-only.
internal/fs: Don't Sync() destination file after copy #1408
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When copying files, directly check for errors when closing
the destination file rather than using out.Sync() with
a deferred Close().
The out.Sync() call can be incredibly expensive on systems which
are I/O-heavy, yet may have more than ample memory and CPU
resources. As part of a file copy, we only need to ensure that
errors writing and closing the file handle are checked and dealt
with, rather than forcing the entire copied file to physical storage.
Environments such as Continuous Integration servers and container
hosts are often I/O-heavy. In such environments, this change
can make a significant difference - milliseconds vs minutes.
What does this do / why do we need it?
Avoid unnecessarily syncing newly copied files to physical storage.
A sync to physical disk imposes heavy latency on completing the file copy, and it is
exacerbated when it is done file-at-a-time.
Replacing the out.Sync() call with an error check on out.Close() can dramatically improve
latency on I/O-heavy machines, and actually improves error detection vs the out.Sync() call.
On a container host that I use for CI, this reduced a
dep
run from 6 minutes to 5 seconds.(the host has copious RAM and CPU, but is heavily I/O bound)
What should your reviewer look out for in this PR?
Do you need help or clarification on anything?
Which issue(s) does this PR fix?