Change project cause Brackets freeze on Linux

[daaru00]

When I change project Brackets GUI freeze and CPU usage goes up to 100% for 30s - 45s, is very bothersome. This problem appears after Brackets upgrade.
I using Brackets 1.5 on Linux OS (Linux Mint 17.2 Cinnamon 64-bit).

[Rynaro]

Every time when you change the project? Or sometimes Brackets changes the project without problems?
In one moment using Brackets v1.4 you faced this problem?

[daaru00]

Unfortunately every time, with v1.4 the project switch was immediate.

[daaru00]

I did a "javascript CPU profiles" with chrome dev-tools during the project loading and seems the function

FileSystemEntry._isWatched

use more resources then other functions.

Maybe this can help you.

[petetnt]

Are you using external dependencies such as node_modules? External libraries can have tens of thousands of files which can be very stressing for the file watchers.

If you are, try installing a file filter such as https://github.com/gruehle/exclude-folders and see if the performance returns to normal.

[daaru00]

Ok, I used https://github.com/zaggino/brackets-file-tree-exclude instead (but is the same things) and the project loading returns immediate, but this is only a workaround.

[abose]

What is the machine configuration [processor,ram etc]?
Can you paste the Help>health report for 1.5 here after you have done the workflows that causes the slowdown and once brackets returns back to normal?

[daaru00]

This is my hardware info:
CPU

Intel(R) Core(TM) i7-3537U CPU @ 2.00GHz (x4)

RAM

6 GB

HDD

SSD 840 evo (reading 540 MB/sec, writing 520 MB/sec)

I open a project and then switch to an other, when the second project is loaded I try to open (for example) file menu. Below I paste the health report and time taken to open that menu.

~20s (brackets-file-tree-exclude disabled)

{
    "uuid": "2d93b812-a410-4400-ab72-f4dedcf537ba",
    "snapshotTime": 1445248859753,
    "os": "linux",
    "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.80 Safari/537.36",
    "osLanguage": "it",
    "bracketsLanguage": "it",
    "bracketsVersion": "1.5.0-16538",
    "fileStats": {
        "openedFileExt": {
            "scss": 2,
            "php": 1,
            "tpl": 3
        },
        "workingSetFileExt": {
            "php": 1,
            "tpl": 1
        }
    },
    "ProjectDetails": {
        "prj2116600247": {
            "numFiles": 8196,
            "cacheSize": 230397178
        },
        "prj224439420": {
            "numFiles": 6382,
            "cacheSize": 39221576
        }
    },
    "projectLoadTimes": ":1:0",
    "fileOpenTimes": ":2:0",
    "ModuleDepsResolved": 30,
    "AppStartupTime": 32,
    "prefNodeSearchDisabled": false,
    "prefInstantSearchDisabled": false,
    "installedExtensions": [
        {
            "name": "brackets-file-tree-exclude",
            "version": "1.0.3"
        },
        {
            "name": "adobe.brackets.extract",
            "version": "0.8.0-1749-release"
        },
        {
            "name": "enturn.quick-search",
            "version": "1.1.4"
        },
        {
            "name": "angularui.angularjs",
            "version": "1.1.2"
        },
        {
            "name": "brackets-angularjs-codehints",
            "version": "0.0.3"
        },
        {
            "name": "brackets-grunt",
            "version": "0.2.7"
        },
        {
            "name": "brackets-materialangularjs-codehints",
            "version": "0.0.2"
        },
        {
            "name": "ivogabe.icons",
            "version": "1.5.5"
        },
        {
            "name": "brackets-beautify",
            "version": "1.2.0"
        },
        {
            "name": "mikaeljorhult.brackets-todo",
            "version": "0.8.1"
        },
        {
            "name": "equals182.eqftp",
            "version": "0.7.0"
        },
        {
            "name": "zaggino.brackets-git",
            "version": "0.14.32"
        },
        {
            "name": "zaggino.brackets-snippets",
            "version": "0.6.1"
        }
    ],
    "bracketsTheme": "dark-theme"
}

~1s (brackets-file-tree-exclude enabled)

{
    "uuid": "2d93b812-a410-4400-ab72-f4dedcf537ba",
    "snapshotTime": 1445249118679,
    "os": "linux",
    "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.80 Safari/537.36",
    "osLanguage": "it",
    "bracketsLanguage": "it",
    "bracketsVersion": "1.5.0-16538",
    "fileStats": {
        "openedFileExt": {
            "scss": 3,
            "php": 1,
            "tpl": 4
        },
        "workingSetFileExt": {
            "php": 1,
            "tpl": 1
        }
    },
    "ProjectDetails": {
        "prj2116600247": {
            "numFiles": 451,
            "cacheSize": 169629568
        },
        "prj224439420": {
            "numFiles": 399,
            "cacheSize": 5235113
        }
    },
    "projectLoadTimes": ":1:0",
    "fileOpenTimes": ":1:1",
    "ModuleDepsResolved": 1,
    "AppStartupTime": 3,
    "prefNodeSearchDisabled": false,
    "prefInstantSearchDisabled": false,
    "installedExtensions": [
        {
            "name": "adobe.brackets.extract",
            "version": "0.8.0-1749-release"
        },
        {
            "name": "enturn.quick-search",
            "version": "1.1.4"
        },
        {
            "name": "angularui.angularjs",
            "version": "1.1.2"
        },
        {
            "name": "brackets-angularjs-codehints",
            "version": "0.0.3"
        },
        {
            "name": "brackets-file-tree-exclude",
            "version": "1.0.3"
        },
        {
            "name": "brackets-grunt",
            "version": "0.2.7"
        },
        {
            "name": "brackets-materialangularjs-codehints",
            "version": "0.0.2"
        },
        {
            "name": "ivogabe.icons",
            "version": "1.5.5"
        },
        {
            "name": "brackets-beautify",
            "version": "1.2.0"
        },
        {
            "name": "equals182.eqftp",
            "version": "0.7.0"
        },
        {
            "name": "mikaeljorhult.brackets-todo",
            "version": "0.8.1"
        },
        {
            "name": "zaggino.brackets-git",
            "version": "0.14.32"
        },
        {
            "name": "zaggino.brackets-snippets",
            "version": "0.6.1"
        }
    ],
    "bracketsTheme": "dark-theme"
}

I have lot of project with node modules or bower components, note the project size ("numFiles": 8196 and "numFiles": 6382) become smaller by excluding some folders using that plugin ("numFiles": 451, "numFiles": 399)

[abose]

can you open debug>open preferences and set a preference "findInFiles.nodeSearch": false and see then try the workflow.
This will disable search index generation. But this will also disable instant search. Please see if that makes any difference.

[daaru00]

Unfortunately no change when disabling that preference.
Perhaps, consider FileSystemEntry._isWatched function use lot of resurces, the problem could be the file change watching?

[abose]

Is this on ext3 filesystem or some other?

[daaru00]

My filesystem type is ext4

[petetnt]

I am pretty confident that the problem lies in the file watchers and other File-related functionalities. Similar ones:

#11777 - Brackets become unresponsive during ember s command
#11733 - Exclude node_modules, bower_components folder option when opening folder
#8188
#6962

etc. etc.

Filtering them out brings back performance but it's not an optimal solution: for example one would want to look into node_modules folders to debug, to find files that are needed to be included, to even do some hotfixing / testing etc. etc.

[daaru00]

Filtering them out brings back performance but it's not an optimal solution: for example one would want to look into node_modules folders to debug, to find files that are needed to be included, to even do some hotfixing / testing etc. etc.

Exactly my concern.

[abose]

What coud be a solution to this issue then?

[petetnt]

I was thinking something along the lines:

• Create a list of common suspects to black-box. This list would include things such as:
  • node_modules
  • bower_components
  • tmp-folders (or temp, or similar)
  • maybe lib/ and vendor/ too.

node_modules is the most important as by design they contain tons of files

• These black-boxed files, which mostly would be external libraries and temporary files, would contains following attributes:
  • They are not being watched for changes by default (filtered away during FileSystem.prototype._watchEntry?
  • They are marked as black-boxed in the folder tree (somehow. how?)
  • Maybe not included in searches either
• black-boxed files can be browsed, opened and modified normally
  • If a black-boxed file is modified, it gains the following attributes
    • It's added to the watched files (/roots)
    • It's added to the searches
  • These additional files could either be: saved until session ends (Brackets is closed) or to ´state.json` and persisted forever.
  • They right clicking an black-boxed file (tree) would offer an option to remote the file (tree) from the blackbox
• Vice versa any folder/file could be black-boxed by right clicking the file (tree)
  • This could either be session long, or persisted by brackets.json
• The list of common suspects could be found and extended in preferences.json

Benefits

• (HUGE) performance benefits (possibly?)
• Ability to browse and modify file structures that cause performance issues within Brackets as they wouldn't have to be excluded in the FileTree anymore

Risks

• Just an concept, could need quite a lot work to perform correctly
• Might create some confusion about some files not being watched / searched
  • Can be avoided with proper introduction to the feature

How does that sound @abose?

[iegik]

Could just read \.*ignore before built tree of project?

[daaru00]

Or a simple setting value with path to ignore in watch is a solution

[abose]

The root causes of the file accesses at project switch time are these:

1. Find in files: This has been now optimized to run in node, and will not block or impact the performance brackets.
2. Code hints: Tern, even though it runs on another thread- fs read events will still have to go through brackets thread, which will block brackets on reading a large number of files.
   One workaround is to move code hints as a node process thus avoiding brackets from doing any fs access- unblocking it. But moving that to a node process will push code into brackets-node. That would be bad for pure web projects like Mozilla Thimble and take significant engineering effort to rewrite the code hinting mechanism in node.
3. file system watchers: If .gitignore Present- ignore it in file watchers/ standard folders[blackbox names suggested by @petetnt] to ignore in file watchers, if the number of files is larger than a particular threshold. I guess just avoiding node_modules& bower_components from file watching would greatly improve the situation without affecting any functionality.

Addressing 2&3 should fix the performance issues without any user input. 3 is easier to implement than 2.

[DavidBruant]

When I change project Brackets GUI freeze and CPU usage goes up to 100% for 30s - 45s, is very bothersome. This problem appears after Brackets upgrade.

Noticing the same problem. It lasts 5 full minutes on my side which is another level of bothersome.
Linux Ubuntu 15.04 64 bits here. ext4 as well.

[daaru00]

@DavidBruant how many file have you project? Can you post the health report?

[DavidBruant]

how many file have you project?

ls -pR | grep -v / | wc -w gives me 19719.
It's a Node.js project with a good amount of node_modules. I do have a .gitignore for it though https://github.com/MyWebIntelligence/MyWebIntelligence/blob/master/.gitignore (it's an open source project)
Ignoring files that should be ignored, around 200 (the above command gives 215 on a fresh git clone of https://github.com/MyWebIntelligence/MyWebIntelligence ), nothing too fancy.

Can you post the health report?

{
    "uuid": "799c27b8-5886-466e-a41c-d861710e58fc",
    "snapshotTime": 1445525163849,
    "os": "linux",
    "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.80 Safari/537.36",
    "osLanguage": "fr",
    "bracketsLanguage": "fr",
    "bracketsVersion": "1.5.0-16538",
    "fileStats": {
        "openedFileExt": {
            "sql": 1,
            "js": 46,
            "md": 1
        },
        "workingSetFileExt": {}
    },
    "ProjectDetails": {
        "prj245341383": {
            "numFiles": 237,
            "cacheSize": 2714276
        }
    },
    "projectLoadTimes": ":1",
    "fileOpenTimes": ":0(0)1[1]:0(0)1[1]:0(1)1[0]:0:0(1)1[1]:1:1:0",
    "ModuleDepsResolved": 1,
    "AppStartupTime": 5,
    "prefNodeSearchDisabled": false,
    "prefInstantSearchDisabled": false,
    "installedExtensions": [
        {
            "name": "fdecampredon.brackets-typescript",
            "version": "0.2.0"
        },
        {
            "name": "gruehle.markdown-preview",
            "version": "1.0.11"
        },
        {
            "name": "brackets-beautify",
            "version": "1.2.0"
        },
        {
            "name": "zaggino.brackets-git",
            "version": "0.14.32"
        },
        {
            "name": "adobe.brackets.extract",
            "version": "0.8.0-1747-release"
        }
    ],
    "bracketsTheme": "light-theme"
}

[petetnt]

@DavidBruant the files aren't actually ignored by Brackets just yet, try one of the extensions mentioned and see if the performance returns :)

[iegik]

Brackets - for developers. Developers must see only code, that they code. There is no reason to include libraries, that are developed in other project.
Also, I found, that opened file automatically have indexed even it is not from current project. That sometimes helps.

So, I see only solution - to exclude any file, that not belongs to the project. (read the .*ignore files)

[abose]

https://github.com/zaggino/brackets-file-tree-exclude is the extension to exclude files from brackets.

[DavidBruant]

try one of the extensions mentioned and see if the performance returns :)

I had missed that part of the discussion. I tried https://github.com/zaggino/brackets-file-tree-exclude , it worked like a charm. Thanks!

[glepretre]

I confirm this issue on Linux Mint Debian 64 bit.

My hardware info if it can help:

Kernel: 3.11-2-amd64 x86_64 (64 bit)
Desktop: MATE 1.6.1
Distro: LinuxMint 1 debian
CPU: Intel Core i7 CPU 860 @1.20 GHz (x4)
HDD: SAMSUNG_HD322GJ (:smirk:)
Memory: 12GB DDR3 (1333 MHz)

Here's my Brackets 1.5 health report:

{
    "uuid": "6d87ccc8-6932-40f3-908f-4efab7b41572",
    "snapshotTime": 1445939747945,
    "os": "linux",
    "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.80 Safari/537.36",
    "osLanguage": "en",
    "bracketsLanguage": "en",
    "bracketsVersion": "1.5.0-16538",
    "fileStats": {
        "openedFileExt": {
            "html": 1,
            "css": 1,
            "js": 4
        },
        "workingSetFileExt": {
            "html": 1,
            "js": 1
        }
    },
    "ProjectDetails": {
        "prj-502147004": {
            "numFiles": 7234,
            "cacheSize": 125832501
        }
    },
    "projectLoadTimes": ":0",
    "fileOpenTimes": ":1",
    "ModuleDepsResolved": 251,
    "AppStartupTime": 253,
    "prefNodeSearchDisabled": true,
    "prefInstantSearchDisabled": true,
    "installedExtensions": [
        {
            "name": "rain.monokai-dark-soda",
            "version": "1.0.8"
        },
        {
            "name": "brackets-ruler",
            "version": "2.3.12"
        },
        {
            "name": "jonwolfe.file-tree-exclude",
            "version": "0.6.3"
        },
        {
            "name": "brackets-paste-and-indent",
            "version": "0.2.0"
        },
        {
            "name": "deddy.additional-right-click-menu",
            "version": "0.5.1"
        },
        {
            "name": "dkehrig.show-whitespace",
            "version": "2.0.1"
        },
        {
            "name": "pessotti.doubleclick-match-brackets",
            "version": "1.0.0"
        },
        {
            "name": "globexdesigns.brackets-jscs",
            "version": "0.7.5"
        },
        {
            "name": "dag0310.tab-out",
            "version": "1.1.0"
        },
        {
            "name": "talmand.select-lines",
            "version": "1.3.0"
        },
        {
            "name": "brackets-add-comment",
            "version": "0.1.2"
        },
        {
            "name": "pretty_json",
            "version": "0.1.2"
        },
        {
            "name": "yellowhangar.guidelines",
            "version": "1.0.1"
        },
        {
            "name": "camden.jshint",
            "version": "2.2.18"
        },
        {
            "name": "brackets-indent-guides",
            "version": "1.3.5"
        },
        {
            "name": "ws-sanitizer",
            "version": "1.2.1"
        },
        {
            "name": "asgerf.bracket-rename",
            "version": "0.2.9"
        },
        {
            "name": "funcdocr",
            "version": "0.8.33"
        },
        {
            "name": "gruehle.markdown-preview",
            "version": "1.0.11"
        },
        {
            "name": "brackets-angularjs-codehints",
            "version": "0.0.3"
        },
        {
            "name": "qw0101.colorhighlighter",
            "version": "1.0.0"
        },
        {
            "name": "edc.brackets-snippets",
            "version": "1.8.1"
        },
        {
            "name": "brackets-vimderbar",
            "version": "0.11.1"
        },
        {
            "name": "nerk.asciidoc-preview",
            "version": "1.1.8"
        },
        {
            "name": "dnbard.extensions-rating",
            "version": "0.7.3"
        },
        {
            "name": "nicolo-ribaudo.brackets-svg-font",
            "version": "0.4.5"
        },
        {
            "name": "ivogabe.icons",
            "version": "1.5.5"
        },
        {
            "name": "ternific",
            "version": "0.9.1"
        },
        {
            "name": "brackets-code-folding",
            "version": "0.3.5"
        },
        {
            "name": "mikaeljorhult.brackets-todo",
            "version": "0.8.1"
        },
        {
            "name": "davidwaterston.goto-matching-bracket",
            "version": "2.1.0"
        },
        {
            "name": "brackets-gitbranch",
            "version": "3.0.6"
        },
        {
            "name": "brackets-sort-text",
            "version": "0.1.4"
        }
    ],
    "bracketsTheme": "rain.monokai-dark-soda"
}

I am in the same case as @daaru00, specially about:

@petetnt:
Filtering them out brings back performance but it's not an optimal solution:
for example one would want to look into node_modules folders to debug,
to find files that are needed to be included, to even do some hotfixing
/ testing etc. etc.

Exactly my concern.

I face the freezing behavior with one of our projects in particular : it is our
root module. It is too big and have too many files bundled I admit but I liked
the way it worked before.

$ ls -pR | grep -v / | wc -w
8844

I have tried setting

"findInFiles.instantSearch": false,
"findInFiles.nodeSearch": false,

in my brackets.json it does speed up a bit but not that much.

I have seen the suggested workaround using extensions to filter/exclude file &
folders from file tree and search but I don't want to rely on third-party
extensions for that, I think performance should be core's reponsibility.

So, I chose to downgrade to 1.4. That's too bad that the 1.5 version have this
performance issue considering what 1.4 brought in this field comparing to 1.3.

I think the black-box concept suggested by @petetnt would fit all my needs:
#11833 (comment)

Ultimately, I wish I could easily and completely disable search indexing,
globally or per project.

NOTE: @daaru00, comparing my installed extensions with yours I noticed that we
both use "mikaeljorhult.brackets-todo" and disabling it or excluding more big
folders into its config speed up Brackets loading. This extension might be
greedy too. But @DavidBruant does not have it.

[petetnt]

@glepretre Yep, brackets-todo is greedy too (as are all other extensions that rely on scraping the project tree (which is a lot of them)).

I agree that performance should be a core responsibility. Excluding the files with an extension is just (or should be just) a stop gap workaround to make the program not completely unusable while these issues are getting fixed 🔍

[abose]

We will be addressing parts of the issues for 1.6. Mainly moving code hints to a different process.
That should bring down the load down to manageable levels.
@swmitra

[abose]

Find in files/ instant search will not significantly affect or block brackets performance with modern machine configurations. We have internally tested instant search scaling well for 1GB+ code bases.

File system watching should be the major resource hog after code hint parsing is moved to node.

[tallandroid]

I like the concept of black-box. Chrome Developer tools has it. We can start with introducing the behavior like .gitignore ?

[petetnt]

@tallandroid Hehe, Chrome DevTools indeed was my inspiration for the black-boxing concept 😎

[tallandroid]

Seeing @zaggino 's code. It basically implements what is needed. All it needs is a UI for it to be a feature. I would let @zaggino chime in.

[zaggino]

Well, exclude folders does the trick, but I'd still prefer to be able to see these folders in the file tree blackened out and click on them to bring them alive :)

[daaru00]

I discovered that exist a limit of indexing file, breackets popup me an error:

I using an Italian version then the error is translated (error during indexing file, reached max limit of file indexed..).
The second error is a git timeout, irrelevant

Now I developing a Magento theme with thousands of files and a brackets plugin to exclude folders it's useless in this case, I can't identify which folder to exclude or not.