"Computing directory structure" sync step doesn't seem to respect project file excludes

[roblg]

The "Computing directory structure" sync step takes 3+ minutes to complete for us. (related: #124, #156). In our case, the issue seems to be caused by the structure of some symlinks within our repo. The short version is that we use https://lernajs.io/ to manage our JS build, and bazel to manage our Java build. Lerna works by symlinking between directories in the repository that have a dependency. Example:

main/
    node_modules/
        foo/
        bar/  --> ../../bar
    bar/
        node_modules/
            baz/ . --> ../../../baz
    baz/
    # etc.

We have a lot of these symlinks. During directory traversal, there end up being really long chains of these dependency symlinks, and the IDE visits every dependency on disk many, many times. We added exclusions for these node_modules dirs to our .bazelproject file, and this works very well -- none of those node_modules directories are indexed (which is fine) which is a huge win for indexing performance. However, the "Computing directory structure" step still seems to be following all those symlinks. If the symlinks are present, the step takes 3+ minutes. If they aren't present, it's done in some reasonable number of seconds.

I couldn't help myself, so I took a poke around DirectoryStructure.java, and it looks like the excluded directories aren't used when doing the traversal for "Computing directory structure"? Based on the suggestion here, it seems like they should be?

I haven't yet verified by stepping through with a debugger that this is the place where the issue is popping up, so it's possible the real issue is somewhere else. I'll figure out how to step through and report back.

[brendandouglas]

We've actually just refactored DirectoryStucture, to multithread the file system operations, and respect excluded directories.

The new plugin build with the fixes should land next Friday.

[roblg]

Y'all are the best. THANK YOU!