Workspace file scanning will throw exceptions on Linux when walking large file trees

[rainerklier]

System Details

• Operating system name and version:
  openSUSE Linux

• VS Code version:
  1.14.0-insider

• PowerShell extension version:
  [email protected]

• Output from $PSVersionTable:
  PSVersion 6.0.0-beta
  PSEdition Core
  BuildVersion 3.0.0.0
  CLRVersion
  GitCommitId v6.0.0-beta.2
  OS Linux 4.11.6-1.ge566a4a-default Create Yeoman generator for PowerShell projects #1 SMP PREEMPT Sat Jun 17 06:45:51 UTC 2017 (e566a4a)
  Platform Unix
  PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
  PSRemotingProtocolVersion 2.3
  SerializationVersion 1.1.0.1

Issue Description

when loading a .ps1 file and editing it and trying to use code-completion and/or suggestion, this simply does not work.
there only shows up a "loading...." popup-message which stays forever.

Attached Logs

in the logfiles i can see a "too many open files" exception.

code-insiders-logs.zip

[rainerklier]

i found out that version 1.2.1 at least works after starting code-insiders.
starting with 1.3.0 it does not work at all any more.

[daviwil]

Wow! That's one I haven't seen before. I think this is due to how we're scanning your workspace's .ps[m]1 files to find function references. Is the number of files in your workspace particularly large?

For some reason the implementation of System.IO.Directory.EnumerateFiles on *NIX machines is triggering Linux's ulimit setting for maximum open file handles:

https://github.com/dotnet/corefx/issues/12503

The suggestion there is to increase your ulimit for open files. Here's how you can do this:

sudo ulimit -n 65536

At any rate, we should be able to catch this error and log it accurately instead of crashing :) I'll fix that.

Relevant stack trace:

    System.AggregateException: One or more errors occurred. (Too many open files) ---> System.IO.IOException: Too many open files
       at System.IO.UnixFileSystem.FileSystemEnumerable`1.OpenDirectory(String fullPath)
       at System.IO.UnixFileSystem.FileSystemEnumerable`1.Enumerate()
       at System.IO.UnixFileSystem.FileSystemEnumerable`1..ctor(String userPath, String searchPattern, SearchOption searchOption, SearchTarget searchTarget, Func`3 translateResult)
       at System.IO.UnixFileSystem.EnumeratePaths(String path, String searchPattern, SearchOption searchOption, SearchTarget searchTarget)
       at System.IO.Directory.EnumerateFileSystemNames(String path, String searchPattern, SearchOption searchOption, Boolean includeFiles, Boolean includeDirs)
       at System.IO.Directory.EnumerateFiles(String path, String searchPattern)
       at Microsoft.PowerShell.EditorServices.Workspace.RecursivelyEnumerateFiles(String folderPath)
       at Microsoft.PowerShell.EditorServices.Workspace.RecursivelyEnumerateFiles(String folderPath)
       at Microsoft.PowerShell.EditorServices.Workspace.RecursivelyEnumerateFiles(String folderPath)
       at Microsoft.PowerShell.EditorServices.Workspace.RecursivelyEnumerateFiles(String folderPath)
       at Microsoft.PowerShell.EditorServices.Workspace.RecursivelyEnumerateFiles(String folderPath)
       at Microsoft.PowerShell.EditorServices.Workspace.RecursivelyEnumerateFiles(String folderPath)
       at Microsoft.PowerShell.EditorServices.Workspace.RecursivelyEnumerateFiles(String folderPath)
       at Microsoft.PowerShell.EditorServices.LanguageService.<FindReferencesOfSymbol>d__18.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.PowerShell.EditorServices.CodeLenses.ReferencesCodeLensProvider.<ResolveCodeLensAsync>d__4.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.PowerShell.EditorServices.CodeLenses.CodeLensFeature.<HandleCodeLensResolveRequest>d__6.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.PowerShell.EditorServices.Protocol.MessageProtocol.MessageDispatcher.<DispatchMessage>d__7.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.PowerShell.EditorServices.Protocol.MessageProtocol.ProtocolEndpoint.<ListenForMessages>d__36.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.PowerShell.EditorServices.Utility.AsyncContext.Start(Func`1 asyncMainFunc, ILogger logger)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
       --- End of inner exception stack trace ---
    ---> (Inner Exception #0) System.IO.IOException: Too many open files
       at System.IO.UnixFileSystem.FileSystemEnumerable`1.OpenDirectory(String fullPath)
       at System.IO.UnixFileSystem.FileSystemEnumerable`1.Enumerate()
       at System.IO.UnixFileSystem.FileSystemEnumerable`1..ctor(String userPath, String searchPattern, SearchOption searchOption, SearchTarget searchTarget, Func`3 translateResult)
       at System.IO.UnixFileSystem.EnumeratePaths(String path, String searchPattern, SearchOption searchOption, SearchTarget searchTarget)
       at System.IO.Directory.EnumerateFileSystemNames(String path, String searchPattern, SearchOption searchOption, Boolean includeFiles, Boolean includeDirs)
       at System.IO.Directory.EnumerateFiles(String path, String searchPattern)
       at Microsoft.PowerShell.EditorServices.Workspace.RecursivelyEnumerateFiles(String folderPath)
       at Microsoft.PowerShell.EditorServices.Workspace.RecursivelyEnumerateFiles(String folderPath)
       at Microsoft.PowerShell.EditorServices.Workspace.RecursivelyEnumerateFiles(String folderPath)
       at Microsoft.PowerShell.EditorServices.Workspace.RecursivelyEnumerateFiles(String folderPath)
       at Microsoft.PowerShell.EditorServices.Workspace.RecursivelyEnumerateFiles(String folderPath)
       at Microsoft.PowerShell.EditorServices.Workspace.RecursivelyEnumerateFiles(String folderPath)
       at Microsoft.PowerShell.EditorServices.Workspace.RecursivelyEnumerateFiles(String folderPath)
       at Microsoft.PowerShell.EditorServices.LanguageService.<FindReferencesOfSymbol>d__18.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.PowerShell.EditorServices.CodeLenses.ReferencesCodeLensProvider.<ResolveCodeLensAsync>d__4.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.PowerShell.EditorServices.CodeLenses.CodeLensFeature.<HandleCodeLensResolveRequest>d__6.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.PowerShell.EditorServices.Protocol.MessageProtocol.MessageDispatcher.<DispatchMessage>d__7.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.PowerShell.EditorServices.Protocol.MessageProtocol.ProtocolEndpoint.<ListenForMessages>d__36.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.PowerShell.EditorServices.Utility.AsyncContext.Start(Func`1 asyncMainFunc, ILogger logger)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)<---

[rainerklier]

what is the workspace?
the folder, where all my source .ps1and .psm1 files are?
if yes, then no, it is not large.
the number of files in my workspace is 8 files and one folder. and in this folder there are 14 files.
and the ulimit for my user is unlimited.
if i call ulimit the result ist unlimited.

[daviwil]

Interesting... If your ulimit is unlimited then you shouldn't be hitting this.

Do you have a folder open in VS Code or just a single file? If you have a folder open, that's what I mean by "workspace".

[rainerklier]

ah, ok.
on the left top side of the VS Code window i see al list of all open files.
on the left bottom side of the VS Code window i see my home-directory, which has of course many files.
i checked, and my home-dir all in all has 683122 files, which is a lot.
how can i force VS Code to have only my development folder open on this left bottom side?
i didn't find a solution to tell VS Code to open only a sub-folder of my home-dir on this left side of the window.

[daviwil]

File -> Open Folder, you can pick your specific development folder this way. The model for VS Code is that you use "Open Folder" to open the specific folder where the files for your project live, that's how it determines your "workspace" which basically equates to your "project".

[rainerklier]

thanks.
this lets me test, if it helps to have only the dev folder open on the left side of the window.

[rainerklier]

at least i am now able to use version 1.4.0 of ms-vscode.PowerShell.
up to now, even this latest version didn't crash.
before, 1.4.0 crashed immediatelly.
so it seems, that it really has something to do with the files section on the left side of the window.
so, it seems, we at least found a workaround.... ;-)

[daviwil]

Glad to hear it's working for you now! I'll fix the crashing issue so that this doesn't happen for someone else :)

[daviwil]

Core PSES issue tracking this is here: PowerShell/PowerShellEditorServices#612