fixes #156 by blacklisting node_modules and bower_components #271

Eonasdan · 2016-04-20T17:55:59Z

fixes #156 by blacklisting node_modules and bower_components folders from FileGlob scan

…rom FileGlob scan

dnfclas · 2016-04-20T17:56:03Z

Hi @Eonasdan, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes. I promise there's no faxing. https://cla2.dotnetfoundation.org.

TTYL, DNFBOT;

dnfclas · 2016-04-20T18:00:01Z

@Eonasdan, Thanks for signing the contribution license agreement so quickly! Actual humans will now validate the agreement and then evaluate the PR.

Thanks, DNFBOT;

upstream

Eonasdan · 2016-04-22T12:15:05Z

I have a client project waiting for this fix before the other developers can use it. Any update on this?

dnfclas · 2016-04-22T12:15:13Z

Hi @Eonasdan, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!

The agreement was validated by .NET Foundation and real humans are currently evaluating your PR.

TTYL, DNFBOT;

vicancy · 2016-04-25T05:32:02Z

Hi @Eonasdan , I am afraid this logic is too hard coded to be part of the glob logic.

For the PathTooLongException for nested node_modules, we also encountered similar issue when using gulp, and we work around the issue by adding the deep nested dependencies to devDependencies section to package.json, as shown in https://github.com/dotnet/docfx/blob/dev/src/docfx.website.themes/default/package.json#L18, hope the info helps.

Eonasdan · 2016-04-25T12:18:11Z

@vicancy I understand the hard coded part.

Could it be read from a config file instead?

<add key="FileGlobBlackList" value="node_modules,bower_components"/>

vicancy · 2016-04-25T14:59:52Z

How about a .globignore as similar to .gitignore? @chenkennt @vwxyzh what's your opinion?

upstream

Eonasdan · 2016-05-02T19:26:47Z

@vicancy @chenkennt @vwxyzh what would like to do with this? As I mentioned, this is currently a blocking issue for one of my client projects.

Also, I haven't done a lot with the new dnx projects. How do I debug the project without running build.cmd, try, fail, build.cmd again? F5 in VS2015 seems to attempt to restore the nuget packages but always fails. I can create a separate issue for this if needed.

chenkennt · 2016-05-03T07:24:48Z

why can't we use exclude to filter node_modules folder? @superyyrrzz to take a look.

superyyrrzz · 2016-05-04T08:15:06Z

@chenkennt exclude doesn't work for now because exception has be thrown before checking excluded files. I create #287 to fix this.

superyyrrzz · 2016-05-06T08:11:13Z

@Eonasdan It's fixed in dev branch in 22dcba0. You can use "exclude": ["node_modules/", "bower_components/"] to avoid the exception.

Eonasdan · 2016-05-06T16:20:50Z

your commit doesn't work for me. Tried it in a test project.

Info: Completed executing in 419.9409 milliseconds.
Error: System.IO.PathTooLongException: The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
   at System.IO.PathHelper.GetFullPathName()
   at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)
   at System.IO.Path.GetFullPathInternal(String path)
   at System.IO.FileSystemEnumerableIterator`1..ctor(String path, String originalUserPath, String searchPattern, SearchOption searchOption, SearchResultHandler`1 resultHandler, Boolean checkHost)
   at System.IO.Directory.GetFiles(String path, String searchPattern, SearchOption searchOption)
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 37
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesCore>d__1.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 29
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Microsoft.DocAsCode.GlobUtility.ExpandFileMapping(String baseDirectory, FileMapping fileMapping) in C:\projects\GitHub\docfx\src\docfx\Glob\GlobUtility.cs:line 31
   at Microsoft.DocAsCode.SubCommands.MetadataCommand.ConvertToInputModel(MetadataJsonItemConfig configModel) in C:\projects\GitHub\docfx\src\docfx\SubCommands\MetadataCommand.cs:line 93
   at Microsoft.DocAsCode.SubCommands.MetadataCommand.<GetInputModels>d__11.MoveNext() in C:\projects\GitHub\docfx\src\docfx\SubCommands\MetadataCommand.cs:line 84
   at Microsoft.DocAsCode.SubCommands.MetadataCommand.Exec(SubCommandRunningContext context) in C:\projects\GitHub\docfx\src\docfx\SubCommands\MetadataCommand.cs:line 34
   at Microsoft.DocAsCode.SubCommands.CompositeCommand.Exec(SubCommandRunningContext context) in C:\projects\GitHub\docfx\src\docfx\SubCommands\CompositeCommand.cs:line 38
   at Microsoft.DocAsCode.Program.ExecSubCommand(String[] args) in C:\projects\GitHub\docfx\src\docfx\Program.cs:line 75


Build failed.
Error: System.IO.PathTooLongException: The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters.
   at System.IO.PathHelper.GetFullPathName()
   at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)
   at System.IO.Path.GetFullPathInternal(String path)
   at System.IO.FileSystemEnumerableIterator`1..ctor(String path, String originalUserPath, String searchPattern, SearchOption searchOption, SearchResultHandler`1 resultHandler, Boolean checkHost)
   at System.IO.Directory.GetFiles(String path, String searchPattern, SearchOption searchOption)
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 37
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesFromSubfolder>d__2.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 51
   at Microsoft.DocAsCode.Glob.FileGlob.<GetFilesCore>d__1.MoveNext() in C:\projects\GitHub\docfx\src\Microsoft.DocAsCode.Glob\FileGlob.cs:line 29
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at Microsoft.DocAsCode.GlobUtility.ExpandFileMapping(String baseDirectory, FileMapping fileMapping) in C:\projects\GitHub\docfx\src\docfx\Glob\GlobUtility.cs:line 31
   at Microsoft.DocAsCode.SubCommands.MetadataCommand.ConvertToInputModel(MetadataJsonItemConfig configModel) in C:\projects\GitHub\docfx\src\docfx\SubCommands\MetadataCommand.cs:line 93
   at Microsoft.DocAsCode.SubCommands.MetadataCommand.<GetInputModels>d__11.MoveNext() in C:\projects\GitHub\docfx\src\docfx\SubCommands\MetadataCommand.cs:line 84
   at Microsoft.DocAsCode.SubCommands.MetadataCommand.Exec(SubCommandRunningContext context) in C:\projects\GitHub\docfx\src\docfx\SubCommands\MetadataCommand.cs:line 34
   at Microsoft.DocAsCode.SubCommands.CompositeCommand.Exec(SubCommandRunningContext context) in C:\projects\GitHub\docfx\src\docfx\SubCommands\CompositeCommand.cs:line 38
   at Microsoft.DocAsCode.Program.ExecSubCommand(String[] args) in C:\projects\GitHub\docfx\src\docfx\Program.cs:line 75
    0 Warning(s)
    1 Error(s)

superyyrrzz · 2016-05-09T03:15:29Z

Hi @Eonasdan , if the folder you want to exclude isn't located in the doc's root path, you should use "exclude": ["**/node_modules/", "**/bower_components/"]

Eonasdan · 2016-05-09T12:47:30Z

Still didn't work. I uploaded a sample project. I removed the node and packages folder to keep the zip light weight so you'll need to restore those of course.

superyyrrzz · 2016-05-09T16:27:43Z

@Eonasdan It's caused by some deep nested package, and it works for me to add "**/node_modules/" to excluded folders like below:

"metadata": [
    {
        "src": [
            {
                "files": [ "**.sln" ],
                "exclude": [
                    "**/obj/**",
                    "**/bin/**",
                    "site/**",
                    "**/node_modules/"
                ]
            }
        ],
        "dest": "obj/api"
    }
  ],

Eonasdan · 2016-05-09T18:17:40Z

I think that managed to be the one node I didn't try for some reason. Go figure. Works with both my test project and my clients project 👍 when do you think this will go into release?

superyyrrzz · 2016-05-10T01:22:41Z

Next Monday, in DocFX 1.9 😃

fixes 156 by blacklisting node_modules and bower_components folders f…

117aca2

…rom FileGlob scan

dnfclas added the cla-required label Apr 20, 2016

dnfclas added cla-signed and removed cla-required labels Apr 20, 2016

Eonasdan and others added 2 commits April 20, 2016 15:11

Merge pull request #3 from dotnet/dev

80e589b

upstream

fix unsynced upstream

19bc5dc

Eonasdan closed this Apr 22, 2016

Eonasdan reopened this Apr 22, 2016

dnfclas added the cla-already-signed label Apr 22, 2016

Merge pull request #5 from dotnet/dev

d4b835d

upstream

chenkennt assigned superyyrrzz May 3, 2016

Eonasdan closed this May 9, 2016

Eonasdan deleted the fix-path-too-long branch May 10, 2016 12:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fixes #156 by blacklisting node_modules and bower_components #271

fixes #156 by blacklisting node_modules and bower_components #271

Eonasdan commented Apr 20, 2016

dnfclas commented Apr 20, 2016

dnfclas commented Apr 20, 2016

Eonasdan commented Apr 22, 2016

dnfclas commented Apr 22, 2016

vicancy commented Apr 25, 2016

Eonasdan commented Apr 25, 2016

vicancy commented Apr 25, 2016

Eonasdan commented May 2, 2016

chenkennt commented May 3, 2016

superyyrrzz commented May 4, 2016 •

edited

Loading

superyyrrzz commented May 6, 2016

Eonasdan commented May 6, 2016

superyyrrzz commented May 9, 2016

Eonasdan commented May 9, 2016

superyyrrzz commented May 9, 2016 •

edited

Loading

Eonasdan commented May 9, 2016

superyyrrzz commented May 10, 2016

fixes #156 by blacklisting node_modules and bower_components #271

fixes #156 by blacklisting node_modules and bower_components #271

Conversation

Eonasdan commented Apr 20, 2016

dnfclas commented Apr 20, 2016

dnfclas commented Apr 20, 2016

Eonasdan commented Apr 22, 2016

dnfclas commented Apr 22, 2016

vicancy commented Apr 25, 2016

Eonasdan commented Apr 25, 2016

vicancy commented Apr 25, 2016

Eonasdan commented May 2, 2016

chenkennt commented May 3, 2016

superyyrrzz commented May 4, 2016 • edited Loading

superyyrrzz commented May 6, 2016

Eonasdan commented May 6, 2016

superyyrrzz commented May 9, 2016

Eonasdan commented May 9, 2016

superyyrrzz commented May 9, 2016 • edited Loading

Eonasdan commented May 9, 2016

superyyrrzz commented May 10, 2016

superyyrrzz commented May 4, 2016 •

edited

Loading

superyyrrzz commented May 9, 2016 •

edited

Loading