shutil: add reflink=False to file copy functions to control clone/CoW copies (use copy_file_range) #81338
Comments
|
bpo-26826 added a new os.copy_file_range() function: As os.sendfile(), this new Linux syscall avoids memory copies between kernel space and user space. It matters for performance, especially since Meltdown vulnerability required Windows, Linux, FreeBSD, etc. to use a different address space for the kernel (like Linux Kernel page-table isolation, KPTI). shutil has been modified in Python 3.8 to use os.sendfile() on Linux: But according to Pablo Galindo Salgado, copy_file_range() goes further: https://bugs.python.org/issue26826#msg344582 Giampaolo Rodola' added: "I think data deduplication / CoW / reflink copy is better implemented via FICLONE. "cp --reflink" uses it, I presume because it's older than copy_file_range(). I have a working patch adding CoW copy support for Linux and OSX (but not Windows). I think that should be exposed as a separate shutil.reflink() though, and copyfile() should just do a standard copy." "Actually "man copy_file_range" claims it can do server-side copy, meaning no network traffic between client and server if *src* and *dst* live on the same network fs. So I agree copy_file_range() should be preferred over sendfile() after all. =) https://bugs.python.org/issue26826#msg344586 -- There was already a discussion about switching shutil to copy-on-write: One problem is that modifying the "copied" file can suddenly become slower if it was copied using "cp --reflink". It seems like adding a new reflink=False parameter to file copy functions to control clone/CoW copies is required to prevent bad surprises. |
vstinner
added
3.9
|
Random notes. Extract of Linux manual page of "cp": -- "Why is cp --reflink=auto not the default behaviour?": -- reflinks are supported by BTRFS and OCFS2. XFS seems to have an experimental support for reflink, 2 years old article: Linux version of ZFS doesn't support reflink yet: -- Python binding using cffi to get reflink: -- "reflink for Windows": "ReFS v2 is only available in Windows Server 2016 and Windows 10 version 1703 (build 15063) or later. -- Linux has 2 ioctl: #include <sys/ioctl.h>
#include <linux/fs.h>
int ioctl(int dest_fd, FICLONERANGE, struct file_clone_range *arg);
int ioctl(int dest_fd, FICLONE, int src_fd);http://man7.org/linux/man-pages/man2/ioctl_ficlonerange.2.html |
|
I'm attaching an initial PoC using FICLONE on Linux and clonefile(3) on OSX. It is also possible to support Windows but it requires a ReFS partition to test against which I currently don't have. I opted for exposing reflink() as a separate function, mostly because:
This initial patch provides a callback=None parameter in case the CoW operation fails because not supported by the underlying filesystems but this is debatable because we can get different errors depending on the platform (which is not good). As such a more generic ReflinkNotSupportedError exception is probably a better choice. |
|
cow.diff: I'm not sure that attempt to call unlink() if FICLONE fails is a good idea. unlink() can raise a new exception which can be confusing. IMHO it's up to the caller to deal with that. Said differently, I dislike the *fallback* parameter of reflink(). Why not exposing clonefile() as os.clonefile() but os._clonefile()? +#if defined(MAC_OS_X_VERSION_10_12) Is Python compiled to target macOS 10.12 and newer? Mac/BuildScript/build-installer.py contains: # $MACOSX_DEPLOYMENT_TARGET -> minimum OS X level
DEPTARGET = '10.5'But I don't know well macOS. "#if defined(MAC_OS_X_VERSION_10_12)" is a check at build time. Does it depend on DEPTARGET? Would it be possible to use a runtime check? You might open a dedicated issue to expose clonefile() since it seems like every tiny detail of this issue is very subtle and should be properly discussed ;-) (I like the idea of exposing native functions like clonefile() directly in the os module!) |
(Oh, my laptop only uses btrfs. Hum, I created a loop device to test an ext4 partition :-)) On an ext4 partition, cp --reflink simply fails with an error: it doesn't fallback on a regular copy. vstinner@apu$ dd if=/dev/urandom of=urandom bs=1k count=1k vstinner@apu$ cp --reflink urandom urandom2 vstinner@apu$ file urandom2 Not only it fails, but it leaves an empty file. I suggest to mimick the Linux cp command: don't automatically fallback (there are too many error conditions, too many risks of raising a new error while handling the previous error) and don't try to remove the created empty file if reflink() fails. |
Me too. A specific exception is better.
Mmm... I'm not sure it's worth it. The only reason one may want to use clonefile() directly is for passing CLONE_NOFOLLOW and CLONE_NOOWNERCOPY flags (the only possible ones):
Good point. It should definitively be loaded at runtime. I will look into that (but not soon). |
|
Adding a new patch (still a PoC, will create a PR when I have something more solid). |
|
I'm curious: is it possible to query the filesystem to check if a copied is copied using CoW? I guess that it's possible, it will be non portable. So I guess that it's better to avoid checking that in unit tests. vstinner@apu$ dd if=/dev/urandom of=urandom bs=1k count=1k vstinner@apu$ stat urandom vstinner@apu$ stat urandom2 Using stat command line tool, I don't see anything obvious saying that the two files share the same data on disk. |
|
See Also: bpo-26826 |
|
Is FICLONE really needed? Doesn't copy_file_range already supports the same? I posted the same question here: https://stackoverflow.com/questions/65492932/ficlone-vs-ficlonerange-vs-copy-file-range-for-copy-on-write-support |
|
I did some further research (with all details here: https://stackoverflow.com/a/65518879/133374). See vfs_copy_file_range in the Linux kernel. This first tries to call remap_file_range if possible. FICLONE calls ioctl_file_clone. ioctl_file_clone calls vfs_clone_file_range. vfs_clone_file_range calls remap_file_range. I.e. FICLONE == remap_file_range. So using copy_file_range (if available) should be the most generic solution, which includes copy-on-write support, and server-side copy support. |
|
As a note, coreutils 9.0 cp defaults now to reflink=auto. https://www.phoronix.com/scan.php?page=news_item&px=GNU-Coreutils-9.0 |
Doesn't this imply to pass some flag to copy_file_range()? "man copy_file_range" says:
How is CoW copy supposed to be done by using copy_file_range() exactly? |
I think copy_file_range() will just always use copy-on-write and/or server-side-copy when available. You cannot even turn that off. |
|
Since Coreutils has I've sketched a patch that adds Diffdiff --git a/Lib/shutil.py b/Lib/shutil.py
index de82453aa5..73417bdafb 100644
--- a/Lib/shutil.py
+++ b/Lib/shutil.py
@@ -43,6 +43,7 @@
# This should never be removed, see rationale in:
# https://bugs.python.org/issue43743#msg393429
_USE_CP_SENDFILE = hasattr(os, "sendfile") and sys.platform.startswith("linux")
+_USE_CP_COPY_FILE_RANGE = hasattr(os, "copy_file_range")
_HAS_FCOPYFILE = posix and hasattr(posix, "_fcopyfile") # macOS
# CMD defaults in Windows 10
@@ -103,6 +104,66 @@ def _fastcopy_fcopyfile(fsrc, fdst, flags):
else:
raise err from None
+def _get_linux_fastcopy_blocksize(infd):
+ """Determine blocksize for fastcopying on Linux.
+
+ Hopefully the whole file will be copied in a single call.
+ sendfile() and copy_file_range() are called in a loop 'till EOF is
+ reached (0 return) so a bufsize smaller or bigger than the actual
+ file size should not make any difference, also in case the file
+ content changes while being copied.
+ """
+ try:
+ blocksize = max(os.fstat(infd).st_size, 2 ** 23) # min 8MiB
+ except OSError:
+ blocksize = 2 ** 27 # 128MiB
+ # On 32-bit architectures truncate to 1GiB to avoid OverflowError,
+ # see bpo-38319.
+ if sys.maxsize < 2 ** 32:
+ blocksize = min(blocksize, 2 ** 30)
+ return blocksize
+
+def _fastcopy_copy_file_range(fsrc, fdst):
+ """Copy data from one regular mmap-like fd to another by using
+ a high-performance copy_file_range(2) syscall that gives filesystems
+ an opportunity to implement the use of reflinks or server-side copy.
+
+ This should work on Linux >= 4.5 only.
+ """
+ try:
+ infd = fsrc.fileno()
+ outfd = fdst.fileno()
+ except Exception as err:
+ raise _GiveupOnFastCopy(err) # not a regular file
+
+ blocksize = _get_linux_fastcopy_blocksize(infd)
+ offset = 0
+ while True:
+ try:
+ n_copied = os.copy_file_range(infd, outfd, blocksize, offset_dst=offset)
+ except OSError as err:
+ # ...in oder to have a more informative exception.
+ err.filename = fsrc.name
+ err.filename2 = fdst.name
+
+ if err.errno == errno.ENOSPC: # filesystem is full
+ raise err from None
+
+ # Give up on first call and if no data was copied.
+ if offset == 0 and os.lseek(outfd, 0, os.SEEK_CUR) == 0:
+ raise _GiveupOnFastCopy(err)
+
+ raise err
+ else:
+ if n_copied == 0:
+ # If no bytes have been copied yet, copy_file_range
+ # might silently fail.
+ # https://lore.kernel.org/linux-fsdevel/[email protected]/T/#m05753578c7f7882f6e9ffe01f981bc223edef2b0
+ if offset == 0:
+ raise _GiveupOnFastCopy()
+ break
+ offset += n_copied
+
def _fastcopy_sendfile(fsrc, fdst):
"""Copy data from one regular mmap-like fd to another by using
high-performance sendfile(2) syscall.
@@ -124,20 +185,7 @@ def _fastcopy_sendfile(fsrc, fdst):
except Exception as err:
raise _GiveupOnFastCopy(err) # not a regular file
- # Hopefully the whole file will be copied in a single call.
- # sendfile() is called in a loop 'till EOF is reached (0 return)
- # so a bufsize smaller or bigger than the actual file size
- # should not make any difference, also in case the file content
- # changes while being copied.
- try:
- blocksize = max(os.fstat(infd).st_size, 2 ** 23) # min 8MiB
- except OSError:
- blocksize = 2 ** 27 # 128MiB
- # On 32-bit architectures truncate to 1GiB to avoid OverflowError,
- # see bpo-38319.
- if sys.maxsize < 2 ** 32:
- blocksize = min(blocksize, 2 ** 30)
-
+ blocksize = _get_linux_fastcopy_blocksize(infd)
offset = 0
while True:
try:
@@ -223,7 +271,7 @@ def _stat(fn):
def _islink(fn):
return fn.is_symlink() if isinstance(fn, os.DirEntry) else os.path.islink(fn)
-def copyfile(src, dst, *, follow_symlinks=True):
+def copyfile(src, dst, *, follow_symlinks=True, allow_reflink=False):
"""Copy data from src to dst in the most efficient way possible.
If follow_symlinks is not set and src is a symbolic link, a new
@@ -264,12 +312,20 @@ def copyfile(src, dst, *, follow_symlinks=True):
except _GiveupOnFastCopy:
pass
# Linux
- elif _USE_CP_SENDFILE:
- try:
- _fastcopy_sendfile(fsrc, fdst)
- return dst
- except _GiveupOnFastCopy:
- pass
+ elif _USE_CP_SENDFILE or _USE_CP_COPY_FILE_RANGE:
+ # reflink may be implicit in copy_file_range.
+ if _USE_CP_COPY_FILE_RANGE and allow_reflink:
+ try:
+ _fastcopy_copy_file_range(fsrc, fdst)
+ return dst
+ except _GiveupOnFastCopy:
+ pass
+ if _USE_CP_COPY_FILE_RANGE:
+ try:
+ _fastcopy_sendfile(fsrc, fdst)
+ return dst
+ except _GiveupOnFastCopy:
+ pass
# Windows, see:
# https://github.com/python/cpython/pull/7160#discussion_r195405230
elif _WINDOWS and file_size > 0:
@@ -402,7 +458,7 @@ def lookup(name):
else:
raise
-def copy(src, dst, *, follow_symlinks=True):
+def copy(src, dst, *, follow_symlinks=True, allow_reflink=False):
"""Copy data and mode bits ("cp src dst"). Return the file's destination.
The destination may be a directory.
@@ -416,11 +472,11 @@ def copy(src, dst, *, follow_symlinks=True):
"""
if os.path.isdir(dst):
dst = os.path.join(dst, os.path.basename(src))
- copyfile(src, dst, follow_symlinks=follow_symlinks)
+ copyfile(src, dst, follow_symlinks=follow_symlinks, allow_reflink=allow_reflink)
copymode(src, dst, follow_symlinks=follow_symlinks)
return dst
-def copy2(src, dst, *, follow_symlinks=True):
+def copy2(src, dst, *, follow_symlinks=True, allow_reflink=False):
"""Copy data and metadata. Return the file's destination.
Metadata is copied with copystat(). Please see the copystat function
@@ -433,7 +489,7 @@ def copy2(src, dst, *, follow_symlinks=True):
"""
if os.path.isdir(dst):
dst = os.path.join(dst, os.path.basename(src))
- copyfile(src, dst, follow_symlinks=follow_symlinks)
+ copyfile(src, dst, follow_symlinks=follow_symlinks, allow_reflink=allow_reflink)
copystat(src, dst, follow_symlinks=follow_symlinks)
return dst
</details> |
On the second thought, I agree with @giampaolo about exposing Note, if we choose to have a separate @giampaolo can I please play with your patches a bit? I want to try to add Windows block cloning support to Also, I am going to create a small pull request to add |
|
Starting by adding a os.reflink() function sounds like a good idea. It sounds less controversial. But again, the most difficult part will be to write the documentation and very clearly describe limitations ;-) |
|
cc @pablogsal |
Do you think it should be in
Right. Also, that is why having something like BTW, please take a look at #93478. |
It depends on how it's implemented. It depends if it fails if a reflink cannot be used, or if it falls back on a regular (slow) copy. Usually, os functions are thin wrapper to syscalls / libc functions, whereas shutil implement fallbacks and are more high level. |
|
@illia-v @vstinner As for where to put openat(AT_FDCWD, "old", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0664, st_size=760, ...}) = 0
openat(AT_FDCWD, "new", O_WRONLY|O_TRUNC) = 4
fstat(4, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
ioctl(4, BTRFS_IOC_CLONE or FICLONE, 3) = 0
//FICLONE file descriptor old(3) to file new(4)https://stackoverflow.com/questions/52766388/how-can-i-use-the-copy-on-write-of-a-btrfs-from-c-code But the idea in python is that I shouldn't have to care about that.
https://docs.python.org/3/library/os.html#os.link |
iritkatriel
added
3.12
|
Pity that this hasn't proceeded further yet, we could use it for |
|
@ThomasWaldmann if this is about Linux, I would strongly recommend considering implementing something on top of os.copy_file_range until this moves forward. While it has some quirks (you need to handle ENOSYS and EXDEV correctly), it's not exactly rocket science either. Even if this moves forward into standard library for 3.12 (which is not a given, there is no agreement on spec), it will still not be generally available for a while. |
|
I'ld need it for as many OSes as possible (Linux, BSDs, macOS, native Windows, OpenIndiana, ...). But thanks for the pointer. Having it on Linux and for py38+ at least would be better than nothing. @nanonyme Hmm, guess |
|
Note also that FreeBSD has copy_file_range(2) in 13+ See: |
|
It's been a few months, and the world of modern filesystems is curious if Python will be joining them? Also, my two cents on the whole "should we reflink by default" business: on a working COW system, having a reflink is ONLY positive. There is no upside to making a full copy; no performance gains later, no loss of safety in the face of modifications, no increased compatibility with other software. Full copies only provide a benefit if we are attempting to backup against partial on-disk corruption. Most of the time, when I make a copy of a file, I don't care about disk corruption; that's what backups are for. Thus, unless you're a backup software, you don't care if you're making a reflink or a copy; the semantics of the two are identical. Even if you are a backup software, reflinking only applies to within the same filesystem; and it goes without saying that two copies of a file on one disk[1] isn't a real backup. Having reflinking not be the default in a high-level language is nuts. Even offering the ability to do a non-reflinked copy is somewhat questionable; if a filesystem claims to support reflinking, it is asserting that there is no semantic difference between files that are duplicated on-disk and ones sharing extents (ie, reflinked). You need filesystem-specific tools or side-channel attacks[2] to even tell if two files are reflinked. reflink=false is a pointless footgun; it risks users confusing reflinks with hardlinks and causing excess disk usage for no reason. [1]: Yes, some filesystems can be on multiple disks. However, a plain copy supplies no guarantee that the file winds up on a different disk, so you're in the same boat. [2]: Kick both files out of the page cache (either by filling it with junk or with madvise()), then measure the time it takes to access each. If the second one is much faster, then it may have been reflinked. Repeat a few thousand times, varying the order, to gain confidence. In my experience, most applications don't care if their file accesses are faster. [edit: apologies, on post-posting proofread, my sass may be a bit strong. Thank you for your work on making great software, even if you haven't managed perfection quite yet] TLDR: Python is a high level langauge, it's insane that it doesn't reflink by default yet. |
erlend-aasland
added
3.13
|
An imperfect attempt to survey how other languages have approached this. While of course not dispositive I think it does suggest that the "free win" of having this be the default behavior is compelling under many different constraints. GCCGolangSince 1.15 io.Copy uses Emacs LispUsed when available. JavaDoes not implement. (Claiming no performance benefit(!?). NodejsLow level support, flags need to be passed along. PHP
RubyUsed by IO.copy_stream when possible. Ruststd::fs::copy and std::io::copy will transparently use facilities like |
Maybe @bplb has not tested on a CoW filesystem? |
Most likely. The Java report has no mention of CoW at all, so if @bplb didn't know that the main benefit of |
|
If memory serves, I tested CoW on BTRFS (for Linux) and on APFS (for macOS) and there was a large performance improvement in both cases. |
|
@cburroughs Yeah, I did my own survey a while ago in https://social.treehouse.systems/@thesamesam/110626072185161018 too. |
|
Even not taking CoW into account, copy_file_range typically avoids a lot of memory copies between userspace and kernel so it's usually overall a win. |
|
Yes, it's especially useful with modern NFS to avoid an unnecessary roundtrip too. |
Also, |
Untrue: implemented in JDK 20, released March 2023 (cf. JDK-8264744). |
Thank you for the correction! JDK looks like another ecosystem that decided on enabling by default and not with a different api. |
Add a `Path.copy()` method that copies the content of one file to another. This method is similar to `shutil.copyfile()` but differs in the following ways: - Uses `fcntl.FICLONE` where available (see GH-81338) - Uses `os.copy_file_range` where available (see GH-81340) - Uses `_winapi.CopyFile2` where available, even though this copies more metadata than the other implementations. This makes `WindowsPath.copy()` more similar to `shutil.copy2()`. The method is presently _less_ specified than the `shutil` functions to allow OS-specific optimizations that might copy more or less metadata. Incorporates code from GH-81338 and GH-93152. Co-authored-by: Eryk Sun <[email protected]>
Add a `Path.copy()` method that copies the content of one file to another. This method is similar to `shutil.copyfile()` but differs in the following ways: - Uses `fcntl.FICLONE` where available (see pythonGH-81338) - Uses `os.copy_file_range` where available (see pythonGH-81340) - Uses `_winapi.CopyFile2` where available, even though this copies more metadata than the other implementations. This makes `WindowsPath.copy()` more similar to `shutil.copy2()`. The method is presently _less_ specified than the `shutil` functions to allow OS-specific optimizations that might copy more or less metadata. Incorporates code from pythonGH-81338 and pythonGH-93152. Co-authored-by: Eryk Sun <[email protected]>
Add a `Path.copy()` method that copies the content of one file to another. This method is similar to `shutil.copyfile()` but differs in the following ways: - Uses `fcntl.FICLONE` where available (see pythonGH-81338) - Uses `os.copy_file_range` where available (see pythonGH-81340) - Uses `_winapi.CopyFile2` where available, even though this copies more metadata than the other implementations. This makes `WindowsPath.copy()` more similar to `shutil.copy2()`. The method is presently _less_ specified than the `shutil` functions to allow OS-specific optimizations that might copy more or less metadata. Incorporates code from pythonGH-81338 and pythonGH-93152. Co-authored-by: Eryk Sun <[email protected]>
Add a `Path.copy()` method that copies the content of one file to another. This method is similar to `shutil.copyfile()` but differs in the following ways: - Uses `fcntl.FICLONE` where available (see pythonGH-81338) - Uses `os.copy_file_range` where available (see pythonGH-81340) - Uses `_winapi.CopyFile2` where available, even though this copies more metadata than the other implementations. This makes `WindowsPath.copy()` more similar to `shutil.copy2()`. The method is presently _less_ specified than the `shutil` functions to allow OS-specific optimizations that might copy more or less metadata. Incorporates code from pythonGH-81338 and pythonGH-93152. Co-authored-by: Eryk Sun <[email protected]>
Use `cp` since `shutil.copy` doesn't do reflinks: python/cpython#81338 reflinks are a lot more efficient. Signed-off-by: Colin Walters <[email protected]>
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: