Support moving across filesystems in pathlib.Path, as shutil.move() does

[LaurentMazuel]

BPO	29805
Nosy	@brettcannon, @pfmoore, @ericvsmith, @tjguk, @zware, @eryksun, @zooba

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-03-13.21:03:44.694>
labels = ['3.8', 'type-feature', 'library', '3.9', '3.10']
title = 'Support moving across filesystems in pathlib.Path, as shutil.move() does'
updated_at = <Date 2021-03-15.21:26:30.922>
user = 'https://bugs.python.org/LaurentMazuel'

bugs.python.org fields:

activity = <Date 2021-03-15.21:26:30.922>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-03-13.21:03:44.694>
creator = 'Laurent.Mazuel'
dependencies = []
files = []
hgrepos = []
issue_num = 29805
keywords = []
message_count = 6.0
messages = ['289549', '289552', '289559', '289630', '289687', '289688']
nosy_count = 8.0
nosy_names = ['brett.cannon', 'paul.moore', 'eric.smith', 'tim.golden', 'Laurent.Mazuel', 'zach.ware', 'eryksun', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29805'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

• GH-73991: Add pathlib.Path.move that can handle rename across FS #30650
• GH-73991: Add pathlib.Path.copy() #119058
• GH-73991: Add pathlib.Path.rmtree() #119060
• [WIP] GH-73991: Add os.copy() and friends #119079
• GH-73991: Split "Directory and file operations" section in shutil docs #119159
• GH-73991: Use same signature for shutil._rmtree_[un]safe(). #120517
• GH-73991: Add follow_symlinks argument to pathlib.Path.copy() #120519
• GH-73991: pathlib ABC tests: add DummyPath.unlink() and rmdir() #120715
• GH-73991: Add pathlib.Path.copytree() #120718
• GH-73991: Support preserving metadata in pathlib.Path.copy() #120806
• GH-73991: Support copying directory symlinks on older Windows #120807
• GH-73991: Support preserving metadata in pathlib.Path.copytree() #121438
• GH-73991: Fix "Operation not supported" on Fedora buildbot. #121444
• GH-73991: Add pathlib.Path.move() #122073
• GH-73991: Rework pathlib.Path.rmtree() into delete() #122368
• GH-73991: Rework pathlib.Path.copytree() into copy() #122369
• GH-73991: Disallow copying directory into itself via pathlib.Path.copy() #122924
• GH-73991: Prune pathlib.Path.delete() arguments #123158
• GH-73991: Add pathlib.Path.copy_into() and move_into() #123314
• GH-73991: Make pathlib.Path.delete() private. #123315
• GH-73991: Prune pathlib.Path.copy() and copy_into() arguments #123337

[LaurentMazuel]

Trying to use Pathlib and Path.replace on Windows if drive are different leads to an issue:

  File "D:\\myscript.py", line 184, in update
    client_generated_path.replace(destination_folder)
  File "c:\\program files (x86)\\python35-32\\Lib\\pathlib.py", line 1273, in replace
    self.\_accessor.replace(self, target)
  File "c:\\program files (x86)\\python35-32\\Lib\\pathlib.py", line 377, in wrapped
    return strfunc(str(pathobjA), str(pathobjB), \*args)
OSError: [WinError 17] The system cannot move the file to a different disk drive: 'C:\\\\MyFolder' -\> 'D:\\\\MyFolderNewName'

This is a known situation of os.rename, and workaround I found is to use shutil or to copy/delete manually in two steps (e.g. http://stackoverflow.com/questions/21116510/python-oserror-winerror-17-the-system-cannot-move-the-file-to-a-different-d)

When using Pathlib, it's not that easy to workaround using shutil (even if thanks to Brett Cannon now shutil accepts Path in Py3.6, not everybody has Py3.6). At least this should be documented with a recommendation for that situation. I love Pathlib and it's too bad my code becomes complicated when it was so simple :(

[LaurentMazuel]

Just to confirm, I was able to workaround it with Py3.6:

    # client_generated_path.replace(destination_folder)
    shutil.move(client_generated_path, destination_folder)

It is reasonable to think about adding a similar feature to pathlib if it doesn't support it and just special-case the drive-to-drive scenario for Windows

[eryksun]

Moving a file across volumes isn't atomic. Getting an exception in this case can be useful. There could be a "strict" keyword-only parameter that defaults to False. If it's true, then replace() won't try to move the file.

[ericvsmith]

I agree this needs to be different from replace(), due to not being atomic. That makes it an enhancement, so I'm removing 3.5.

I'm +1 on Path supporting something like shutil.move().

[brettcannon]

I also support the idea of getting something like shutil.move() into pathlib.

[brettcannon]

I should also mention that rename() (https://docs.python.org/3/library/pathlib.html#pathlib.Path.rename) and replace() (https://docs.python.org/3/library/pathlib.html#pathlib.Path.replace) already do exist, so it might be best to add a keyword-only flag to one of those for this use-case.

[barneygale]

I'm also +1 on adding pathlib.Path.move(). However I'd prefer we move the shutil implementation into pathlib, as proposed here. The implementation in shutil would become a stub that calls into pathlib, roughly:

def move(src, dst, copy_function=copy2):
    return pathlib.Path(src).move(dst, copy_function=copy2)

I'm interested in this approach because I'm hoping to add a pathlib.AbstractPath class in future. Users would be able to subclass AbstractPath, and by implementing a small handful of low-level abstract methods like stat(), open() and iterdir(), benefit from high-level methods like move() in their objects. This would work even if the source and destination are different kinds of path (e.g. a local filesystem path and a path in a tar archive).

To achieve this we'd need to make pathlib accept bytes paths, as shutil does. Personally I'm fine with this - it's not much work and perfectly reasonable.

We'd also need to move other shutil functions that move() relies on to pathlib, notably the copy*() functions, excluding copyfileobj() and friends.

On rename() / replace(): I'd like these to call through to move(atomic=True). IIRC these methods are identical on POSIX and only differ slightly on Windows. That's a wrinkle we should smooth out in pathlib IMO.

[zooba]

I'd prefer we move the shutil implementation into pathlib

Sounds reasonable to me, though would we want to include copy_function in the API? It seems like an opportunity to say that Path is opinionated about the "most move-like way to copy", and if you need to override that then handle it yourself or stick with shutil.

[ericvsmith]

I think @barneygale 's plan is sound.

[pfmoore]

I agree with @zooba, exposing the copy_function in the pathlib API seems incompatible with the general style of pathlib. Otherwise the plan looks good.

[barneygale]

I agree about copy_function. Perhaps we could do something like this?

    def copy(self, dst, *, follow_symlinks=True, copy_mode=True, copy_stat=False):
        ...

    def move(self, dst, *, atomic=False, **kwargs):
        if atomic:
            return os.rename(self, dst)
        self.copy(dst, **kwargs)
        self.unlink()

[zooba]

I don't like the atomic argument, and especially its default. Implementations should get to implement it as efficiently as possible without needing to respect that kind of argument, and callers shouldn't expect that level of control over semantics.

For example, on Windows we explicitly disallow os.rename from moving files to other drives, because it wouldn't match our documented behaviour. But a new WindowsPath.move function should be able to allow that, and take advantage of the OS optimisations (yes, they exist!) for moving a file rather than copying it across drives.

If we were to define semantics for atomic, the only way to allow this would be "atomic=True will try to move, and atomic=False won't bother trying to move", at which point why would anybody ever pass False? Especially since there's no back-compat here, and so code on older versions will have to stick with try: ...rename ... except: ... copy.

We should make these functions an improvement in semantics, not just discoverability, and ideally more appropriate for whatever platform they're being used on than the POSIX semantics inherent to posixmodule.

So I think the only option we want on move is follow_symlinks=True,¹ and the method "does whatever is necessary to move the file or directory as efficiently as possible into the new location while matching the semantics of an in-place rename" (roughly, copy_mode=True, copy_stat=True)

Similarly, I think we should simplify copy down to only have follow_symlinks=True,² and the method "does whatever is necessary to copy the file or directory as efficiently as possible into the new location while matching the semantics of a newly created file" (roughly, copy_mode=some bits, copy_stat=some bits - and this one is why the vague definition is important 😉 )

Footnotes

1. Where follow_symlinks=False is carefully defined to mean "if self is a link, it moves the link unless that would cause the new link to fail to refer to the same target as the original link, in which case the target is copied into the new location and the original link is removed". ↩

2. I think the same definition as for move works here, but I haven't thought through this one as thoroughly. ↩

[barneygale]

My example was a bit rubbish - here's a corrected version:

    def copy(self, dst, *, follow_symlinks=True, copy_mode=True, copy_stat=True):
        ...

    def move(self, dst, *, atomic=False, **kwargs):
        try:
            return os.rename(self, dst)
        except OSError:
            if atomic:
                raise  # Move can't be achieved atomically
        self.copy(dst, **kwargs)
        self.unlink()

So move(atomic=True) would call os.rename() only and allow exceptions to propagate, whereas atomic=False will try a rename, and failing that use a copy/delete. I think there's common use cases for both.

I think I agree with you on removing copy_mode and copy_stat.

[barneygale]

I suppose one could argue that folks should just use os.rename() if they want an atomic rename! :)

[zooba]

I suppose one could argue that folks should just use os.rename() if they want an atomic rename! :)

Yeah, I'll argue that one ;)

We'll have to explain that move may result in both old and new names existing simultaneously while the move takes place, depending on the operating system and file systems in use. That's a perfectly fine place to say that if you'd rather that not happen, and are willing to accept exceptions if it's unavoidable, then you should use rename.

[barneygale]

Should we deprecate pathlib.Path.rename() and replace() when we add move()?

[barneygale]

Do we even need a copyfile() function, if copy() will do the same?

Good point - perhaps not!

[barneygale]

It's done! Thank you everyone who helped formulate and review this feature.

We've added new copy(), copy_into(), move() and move_into() methods, which work on both files and directories. The interfaces are refined to pathlib standards, with certain nonessential functionality (like support for filtering in copy()) omitted for now. Users may wish to propose new capabilities on the ideas forum.

The new methods expect exact targets (or target directories for the _into() variants), whereas shutil checks for existing directories and adjusts the target as necessary. The latter behaviour is helpful in an interactive shell with tab-complete, but less so in a stored program where state of the filesystem might not be known (for example, shutil.copy('a', 'b') could create either b or b/a).

The copy() and copy_into() methods use fcntl.FICLONE or os.copy_file_range where available (see GH-81338, GH-81340). This work has stalled for years in shutil for backwards-compatibility concerns, happily not present here.

Feedback most welcome! We have ~9 months until 3.14 beta 1 where the interface is more locked down, so there's still plenty of time make changes. I'll close this issue, but I can re-open it (or log new issues) if problems are identified. Thanks again, all.