freeze due to name collision in module loading on case-insensitive filesystems

[SimonDanisch]

Very weird bug I found and I couldn't work out a minimal version to reproduce it.
It happens on windows, but not linux :

Julia Version 0.3.2+2
Commit 6babc84* (2014-10-22 01:21 UTC)
Platform Info:
  System: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-4200U CPU @ 1.60GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

and

Julia Version 0.4.0-dev+1603
Commit b923a43* (2014-11-14 11:57 UTC)
Platform Info:
  System: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-4200U CPU @ 1.60GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

Its about GLPlot and runtest.jl which I could reduce to only one line, namely using GLPlot.
I'm using GLPlot, GLAbstraction, ModernGL, GLText, GLWindow and Reactive on master.
When I run runtest.jl either via Pkg.test("GLPlot"), or directly with julia runtests.jl, it hangs in ~95% of the cases at:
Images/src/libmagicwand.jl/ line 11 import Images: (@Dict)
or in ~5% of the cases at:
ModernGL/src/glconstants.jl line 1 @GenEnums begin
If I put the file runtest.jl anywhere but the test folder, it works in 99.9% of the cases. I don't know where it stops in 0.1% of the cases, since it didn't hang anywhere else since I started investigating.
This behavior is known to me since approximately Mai, but it hardly happened, so I assumed it's a bug from my side...

[SimonDanisch]

Okay, one oddness less:
It was about images.jl in the test folder.
But still, why doesn't this happen on linux, why isn't there any error message, why did it stop randomly in another location, and what about the other 0.1%, where it also hangs started from anywhere (without images.jl around).
Is using on purpose not case sensitive (at least concerning the file name)?

[stevengj]

Assuming you are using NTFS, my understanding is that whether file-access is case-sensitive depends upon the filesystem API we are using. Since libuv uses the Win32 API, it is case-insensitive.

In contrast, Linux (and other traditional Unix) filesystems are always case-sensitive.

(Note that Mac filesystems are also case-insensitive by default. I actually had the same problem recently where using Blosc froze on my Mac because I had a blosc.jl file in the current directory.)

[SimonDanisch]

Oh I wouldn't have expected that... Changing images.jl to Images.jl actually does introduce the same behaviour on Ubuntu...
Isn't this something julia should abstract away? As it obviously can introduce some weird bugs, which are than platform dependent...

[stevengj]

I don't think it's practical to abstract this away (except by making every system act case-insensitive), when it can be an intrinsic property of the filesystem (e.g. in MacOS or Windows FAT).

[SimonDanisch]

Well, it'd be nice to just have one way of doing this, even if that means its case sensitive everywhere. After all, one has to program this way anyways, as your scripts are supposed to run cross-platform with Julia. So opening files on windows ignoring cases is not really an option, if you want to share your code.
But I guess it'd would make it harder to work with programs using the Win32 API, so its not going to happen!?

[stevengj]

This arises in just about every programming language, not just Julia. The usual advice is to be consistent in what capitalization you use when referring to a file, and to avoid having two files that differ only in their capitalization.

I don't think it's possible to make things case sensitive everywhere, because the case is lost when you store the file on a case-insensitive filesystem like FAT; on the other hand, you can theoretically still access the case on case-preserving case-insensitive filesystems (like Mac and Windows). It would be technically possible to make Julia case insensitive everywhere, but then this is both ugly and potentially slow (because you have to walk the filesystem directory yourself, and decide how to resolve case conflicts).

Actually, on case-insensitive case-preserving systems (such as MacOS and Windows NTFS/Win32), it sounds like Python actually does walk the directory path itself to enforce case-sensitivity in module loading. See Python PEP 235. I suppose we could do this too.

[SimonDanisch]

I was afraid so...

[StefanKarpinski]

One possible approach migh be to open the file via the fs api and then once it's open check its name and see if that name matches the expected case. This is slow but would be ok in the specific case of loading code.

[SimonDanisch]

That sounds like a good solution for "high risk" areas.
Besides the case issue, shouldn't it at least throw a warning, if it loads a file via using which doesn't contain a module with the name and even overwrites an existing official module?
I'm not a big fan of warnings, but in this case, I don't really see how this should be a widely used use-case. Maybe it's even reasonable to throw an error...
Also I might run this via a script a few more times, to hunt down the remaining 0.1% which where independent from this problem and randomly happened before.

[stevengj]

@StefanKarpinski, what you suggest used to be what Python did, but then they changed it in PEP 235. I think the issue is that this "open then check" approach depends on which file happens to appear first in the path. It doesn't seem like a bad compromise to me, but it might be worth looking back at the Python discussions circa 2001 for why they opted to walk the directory path instead.

[jakebolewski]

Closed by #13542

[stevengj]

@jakebolewski, I don't think this is fully resolved. The issue above was about include, and include was not affected by #13542.

If we want to fully address this, we need a followup PR that uses isfile_casesensitive in include_from_node1. cc @malmaud.

[stevengj]

Maybe this can be closed?

My feeling is that we should be case-sensitive (i.e. ignore the filesystem semantics) in things like using Foo and Pkg.add("Foo") where Foo is a semantic Julia object and the file is an implementation detail. That's addressed by #13452 and #18210.

On the other hand, in a function like open or include, where you actually pass a pathname, my inclination is to respect the filesystem semantics (including case insensitivity). People on case-insensitive filesystems need to know that filenames are insensitive.

The original issue here, with an import statement being confused by a local file, was fixed in #12695 now that import only looks in the global path.

[SimonDanisch]

Fine by me ;) My concern from this issue is fixed after all!