Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add function to make paths absolute, which is different from canonicalization #59117

Open
retep998 opened this issue Mar 11, 2019 · 14 comments
Open
Labels
C-feature-request Category: A feature request, i.e: not implemented / a PR. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.

Comments

@retep998
Copy link
Member

retep998 commented Mar 11, 2019

Many users want to turn relative paths into absolute paths, but the only tool that libstd currently offers is canonicalize which is bad because it has to hit the filesystem which has several downsides:

  1. It takes time to hit the filesystem.
  2. The path has to actually exist.
  3. It fails on certain types of drives such as RAM drives on Windows.
  4. The path it creates is a \\?\ path which not all software can handle correctly and imposes requirements on further path manipulation that few users are even aware of.

Needing symbolic links actually resolved is an extremely rare use case, and is often misused to compare paths for equality when in reality you should be comparing file IDs due to things like hard links existing.

A new function (normalize or make_absolute or something, bikeshed away) should be added that will turn a relative path into an absolute path without touching the filesystem. On Windows this should either call GetFullPathNameW or do the pure Rust equivalent while on unixy platforms... something should happen, I have no idea.

If such a function did exist, rustc could start using that instead of canonicalize which would fix a whole host of issues including:
#74327
#74146
#59107
#58613
#55812
#52440
#48249
#45067
#42869

@retep998 retep998 added the T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. label Mar 11, 2019
@Mark-Simulacrum Mark-Simulacrum added the C-feature-request Category: A feature request, i.e: not implemented / a PR. label Mar 12, 2019
@Mark-Simulacrum
Copy link
Member

Semi-related issues/PRs: #47402, #47363, and rust-lang/rfcs#2208.

@retep998
Copy link
Member Author

One big question for the libs team to decide on is how to handle .. and . on linux. On Windows C:\foo\..\bar is equivalent to C:\bar and both paths will always point to the same thing. On linux however it is possible for /foo/../bar to not point to the same thing as /bar. Given a current directory of /foo and a relative path of ../bar, should normalize on linux return /foo/../bar or /bar?

@Xanewok
Copy link
Member

Xanewok commented Apr 9, 2019

Some prior art might be the Node path.normalize function which (it seems?) doesn't hit the FS and just tries to do the reasonable thing given path structure, e.g.:

For example, on POSIX:

path.normalize('/foo/bar//baz/asdf/quux/..');
// Returns: '/foo/bar/baz/asdf'

On Windows:

path.normalize('C:\\temp\\\\foo\\bar\\..\\');
// Returns: 'C:\\temp\\foo\\'

petrutlucian94 added a commit to petrutlucian94/docker-buildkite-plugin that referenced this issue May 8, 2019
As a quick workaround for a rustc bug [1], we'll copy the project dir,
avoiding using the mounted dir directly (which rustc cannot handle).
I'm doing this at the buildkite plugin level to avoid polluting the
CI pipeline.

[1] rust-lang/rust#59117
@nayeemrmn
Copy link

A function called normalize() will not be expected to convert to an absolute path, that name should be used for rust-lang/rfcs#2208.

@nayeemrmn
Copy link

nayeemrmn commented Dec 6, 2019

I guess fs::canonicalize() is like realpath -e and the requested feature is like realpath -ms on Linux.

I want to reiterate that fs::normalize() should not be used for this, but for rust-lang/rfcs#2208 which is different from both of these. It doesn't account for CWD and leaves relative paths relative, making it the proper equivalent of Node's path.normalize() referenced above.

@Ciantic
Copy link

Ciantic commented Jan 12, 2021

I think this should be mentioned here too, @dylni has created a normpath crate to do pretty much this: https://crates.io/crates/normpath it seems to also call the GetFullPathNameW on Windows, and doesn't hit the disk.

@dylni
Copy link
Contributor

dylni commented Jan 13, 2021

Thanks @Ciantic!

To clarify, PathExt::normalize only doesn't touch the filesystem on Windows. On Unix, it needs to canonicalize the path to resolve it correctly. Canonical Unix paths don't have the same issues as verbatim paths on Windows.

PathExt::normalize_virtually from that crate can be used to normalize paths without touching the filesystem, but PathExt::normalize is usually what you want.

@MouseProducedGames
Copy link

"It fails on certain types of drives such as RAM drives on Windows."

Being able to compile on a RAM drive on Windows, would save a lot of SSD thrashing.

@ChrisDenton
Copy link
Member

ChrisDenton commented May 7, 2021

@MouseProducedGames Note that it only fails on improperly implemented RAM disk software (unfortunately this includes at least one rather popular one). Something well made, like Radeon RAMDisk, should work fine.

@MouseProducedGames
Copy link

@MouseProducedGames Note that it only fails on improperly implemented RAM disk software (unfortunately this includes at least one rather popular one). Something well made, like Radeon RAMDisk, should work fine.

That is good to know; and thanks for the recommendation.

matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Jan 11, 2022
…imulacrum

`std::path::absolute`

Implements rust-lang#59117 by adding a `std::path::absolute` function that creates an absolute path without reading the filesystem. This is intended to be a drop-in replacement for [`std::fs::canonicalize`](https://doc.rust-lang.org/std/fs/fn.canonicalize.html) in cases where it isn't necessary to resolve symlinks. It can be used on paths that don't exist or where resolving symlinks is unwanted. It can also be used to avoid circumstances where `canonicalize` might otherwise fail.

On Windows this is a wrapper around [`GetFullPathNameW`](https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfullpathnamew). On Unix it partially implements the POSIX [pathname resolution](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_13) specification, stopping just short of actually resolving symlinks.
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Jan 11, 2022
…imulacrum

`std::path::absolute`

Implements rust-lang#59117 by adding a `std::path::absolute` function that creates an absolute path without reading the filesystem. This is intended to be a drop-in replacement for [`std::fs::canonicalize`](https://doc.rust-lang.org/std/fs/fn.canonicalize.html) in cases where it isn't necessary to resolve symlinks. It can be used on paths that don't exist or where resolving symlinks is unwanted. It can also be used to avoid circumstances where `canonicalize` might otherwise fail.

On Windows this is a wrapper around [`GetFullPathNameW`](https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfullpathnamew). On Unix it partially implements the POSIX [pathname resolution](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_13) specification, stopping just short of actually resolving symlinks.
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Jan 12, 2022
…imulacrum

`std::path::absolute`

Implements rust-lang#59117 by adding a `std::path::absolute` function that creates an absolute path without reading the filesystem. This is intended to be a drop-in replacement for [`std::fs::canonicalize`](https://doc.rust-lang.org/std/fs/fn.canonicalize.html) in cases where it isn't necessary to resolve symlinks. It can be used on paths that don't exist or where resolving symlinks is unwanted. It can also be used to avoid circumstances where `canonicalize` might otherwise fail.

On Windows this is a wrapper around [`GetFullPathNameW`](https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfullpathnamew). On Unix it partially implements the POSIX [pathname resolution](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_13) specification, stopping just short of actually resolving symlinks.
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this issue Jan 13, 2022
…imulacrum

`std::path::absolute`

Implements rust-lang#59117 by adding a `std::path::absolute` function that creates an absolute path without reading the filesystem. This is intended to be a drop-in replacement for [`std::fs::canonicalize`](https://doc.rust-lang.org/std/fs/fn.canonicalize.html) in cases where it isn't necessary to resolve symlinks. It can be used on paths that don't exist or where resolving symlinks is unwanted. It can also be used to avoid circumstances where `canonicalize` might otherwise fail.

On Windows this is a wrapper around [`GetFullPathNameW`](https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfullpathnamew). On Unix it partially implements the POSIX [pathname resolution](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_13) specification, stopping just short of actually resolving symlinks.
bors added a commit to rust-lang-ci/rust that referenced this issue Feb 13, 2022
…ulacrum

`std::path::absolute`

Implements rust-lang#59117 by adding a `std::path::absolute` function that creates an absolute path without reading the filesystem. This is intended to be a drop-in replacement for [`std::fs::canonicalize`](https://doc.rust-lang.org/std/fs/fn.canonicalize.html) in cases where it isn't necessary to resolve symlinks. It can be used on paths that don't exist or where resolving symlinks is unwanted. It can also be used to avoid circumstances where `canonicalize` might otherwise fail.

On Windows this is a wrapper around [`GetFullPathNameW`](https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfullpathnamew). On Unix it partially implements the POSIX [pathname resolution](https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_13) specification, stopping just short of actually resolving symlinks.
@andylizi
Copy link
Contributor

Should this be closed now that #91673 is merged?

@gyk
Copy link

gyk commented Feb 26, 2023

It fails on certain types of drives such as RAM drives on Windows.

This appears to be fixed in the latest Windows 11 updates. I have tested calling canonicalize on (improperly implemented) ramdisk and network drive, and it no longer returns "Incorrect function". Could anyone verify it?

@ChrisDenton
Copy link
Member

@gyk Do you have a version number? 10.0.22621 still fails last I checked (e.g. with ImDisk). Though I admit I haven't checked the latest Beta or Preview channels.

@gyk
Copy link

gyk commented Feb 26, 2023

@ChrisDenton The OS builds are 22623.1325 (beta channel) and 22621.1105. The ramdisk software I tested was Ultra RAMDisk Lite (IIRC it didn't work before?), I just tried it again with ImDisk and yes, it still failed. And canonicalize now works on network drives mapped to QNAP or Synology NAS and I'm certain it used to return IO errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-feature-request Category: A feature request, i.e: not implemented / a PR. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests

10 participants