-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Hardlinks not correctly detected on SMB network share (NTFS?) #3080
Comments
@rxxg I am unabe to reproduce the issue,
Does the status display corrupted cache WARNING? [EDIT] [EDIT2] |
@rxxg Also, as a temporary workaround you can change cache type to copy ( |
@pared Thanks for looking at this. Yes, the use of a network drive (repo and cache) is essential. (I had been using copy cache but it is very slow for our use case since the copy that DVC does involves reading then writing to the network drive in 16k chunks. Native Windows copy is of the order of 3 seconds for a 128Mb file, 30 seconds for shutil.copyfileobj. There may be a separate bug report or PR for this.) |
For the record:
|
@rxxg Ok, thank you very much, I am trying to reproduce it on my machine. |
@rxxg Could you please install psutil with Side note for us: need to improve the way |
Sure.
|
@rxxg I see that it is reporting NTFS, but you were saying you are on NFS. Was it a typo or am I missing something? |
Sorry, typo 😳 Windows NTFS network share |
@rxxg Thanks for clarifying! Makes more sense now. Btw, I suppose you don't have WSL enabled either, right? That would explain why |
The issue might be caused by us using GetFileInformationByHandle, which could return incomplete data https://stackoverflow.com/questions/3523271/get-windows-hardlink-count-without-getfileinformationbyhandle. Looks like FindFirstFileNameW and FindNextFileNameW are the alternatives. And ansible is actually using it as well https://github.com/ansible/ansible/blob/105f60cf480572fb5547794cda1f9a05559ae636/lib/ansible/module_utils/powershell/Ansible.ModuleUtils.LinkUtil.psm1#L230 . |
fsutil does work as expected on my local drive. I don't know what WSL is I'm afraid. |
So we need to make our |
@rxxg Created a POC patch for it. Please run
to install it and then run
and share its output. |
Bad news 😞
|
@rxxg Could you show |
Oops, sorry.
|
Thanks @rxxg ! Interesting. Btw, are you aware of how the network share is setup? I'm not really a windows guy, and google didn't help much 🙁 Is there a central server that you are connected to? If that is so, my only explanation right now is that it is running something old which doesn't support Looks like we ran out of options here, fs is returning an incomplete data and alternative ways of counting links are not supported. Another option that might work for you is enabling symlink support on your machine and using |
I don't have many details about the network server, sorry. Windows tells me that there is a cluster running NTFS + DFS, but I don't know what's on the other side. My biggest concern at this point is that DVC is detecting that hardlinks are available (which they are, kind of) and trying to use them, but then failing to detect that the links have been correctly created. So if there are no other options for checking links DVC should refuse to try and create them and fallback to the next cache type? I will try symlinks next. [EDIT] |
Yes, will update my PR to do preciselly that. Currently using simple asserts in it, but should actually rise a proper exception instead. Thanks for the reminder! 🙂
Have you tried installing our exe? Or do you have very limited rights on your machine? |
I'm working on a utility which combines git + DVC in one UI for our particular workflow, to be internally redistributed as one package. My user base doesn't have admin rights on their machines. (I'm assuming the dvc exe needs admin to set up symlinks?) |
I've edited the issue description, I'm not 100% convinced that there actually is an NTFS system on the other side of the network. Windows seems to always report NTFS in the UI, even when there is something else (like an OSX share). |
Yes, but I think there is a way to do that without those rights https://www.google.com/search?q=windows+symlink+without+admin , but I haven't tried that myself 🙁 Or, well, you could ask your admin to enable those for your machines 🙂 |
Using DVC version 0.80, with repo and cache both on the same network drive.
Windows setup.
I'm testing the
unprotect
command, but I assume other commands relying on the same check will also exhibit strange failures.As you can see, I checked out my files from git at 14:57 today; dvc checkout created a protected link to the original 128Mb file created 10 days ago, rather than a copy (as requested), but Windows can't tell me that it is a link. Trust me, it is 😉
Strange - the
unprotect
operation was very fast, and the creation date of the file is still last year rather than today.Sure enough, modifying
big_file
corrupts the cache, since I was not modifying a copy.The text was updated successfully, but these errors were encountered: