-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cpptools extension using excessive memory in 0.28.1 in linux #5602
Comments
I assume it repros with 0.28.2 too? We considered the 0.28.1 changes to be small in scope and "safe" -- I'll double check for anything that could cause this... |
Can you confirm whether the memory usage is from cpptools or the child cpptools-srv processes? So far we're unable to repro the issue and don't see any change from 0.28.1 that could be causing this. |
Also, is stuff otherwise functioning or does there appear to be a deadlock or other issue? What actions appear to be causing the memory usage to rise? Does it rise if you just open 1 .cpp file or does it gradually rise the more files that are open? Are edits or other actions required? |
0.28.1 fixed a high hitting IntelliSense crash with the cpptools-srv process -- so now it could be hitting a new memory issue that was not hittable previously due to the blocking crash. The tool you're using to view memory usage may be reporting the child cpptools-srv process memory usage under cpptools. If you close all the editors, does the memory free up? Does setting the C_Cpp.intelliSenseEngine setting to "Tag Parser" fix the memory issue (that disables the cpptools-srv processes from launching, not recommended due to reduced functionality). |
Hi Sean,
Thanks for the prompt response.
I believe it was from the child cpptools-srv processes. One particular
thing. Shutting down VSCode and restarting did not fix it. I had to
manually kill the processes.
There was no deadlock, but there was significant system perf degradation as
it was constantly paging in and out of the swap partition.
My regular use is opening the root folder that has a tree with around 5000+
files and continuing to work in that. I never close a file that I open. My
editor is set to not reuse a tab. i.e. every file I open via the quick open
action goes into a new tab unless its already available in a tab. The
number of tabs range from 200-500 in regular use. I use intellisense a lot
(2 features in particular, header/source file flip and go to definition). I
use the timeline feature at times and look at diffs. I also use source
control integration to look at diffs, and sometimes to stage files.
When using it takes around a few hours before I start to see the
degradation of performance which is usually a lot of hitches and stalling
system wide.
I don't think closing editors frees up memory. I tried that at some point.
In fact, closing VS Code leaves dangling cpptools and cpptools-srv. I can
later try the tag parser and see if that helps.
Is there anyway to setup a log file or status dump that I can send your way?
- Shammi
…On Wed, Jun 3, 2020 at 4:13 PM Sean McManus ***@***.***> wrote:
0.28.1 fixed a high hitting IntelliSense crash with the cpptools-srv
process -- so now it could be hitting a new memory issue that was not
hittable previously due to the blocking crash. The tool you're using to
view memory usage may be reporting the child cpptools-srv process memory
usage under cpptools.
If you close all the editors, does the memory free up? Does setting the
intelliSenseEngine to "Tag Parser" fix the memory issue (that disables the
cpptools-srv processes from launching).
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#5602 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABOLYVLWR5MKCQQ2EUMWTMLRU3KLBANCNFSM4NQ7HHZA>
.
|
The normally expected behavior is that on startup, one cpptools-srv will be launched for the active file, and then as you switch to other open documents, additional cpptools-srv should spawned for each new TU (.cpp file) up to your logical processor count, afterwhich the oldest used cpptools-srv should be closed -- do you see that happening or does the cpptools-srv count keep increasing endlessly? Closing VS Code is supposed to kill the cpptools process and the cpptools-srv process -- does the dangling process issue occur every time for you? We've heard other users hitting this after upgrading their extension, but if it's happening more often then that's a new issue. Using the "Tag Parser" should fix the memory issue, but it's not recommended since the features you use may not work (i.e. only global symbols will be available with no TU-based info). |
So I installed 0.28.2 today and restarted.
I notice 4 cpptools-srv processes open each taking up max of 200 MB
However cpptools process is the culprit. I just reloaded VSCode (in order
to reactivate updated cpptools extension) and since then its been
constantly chewing up memory. I can see it increase around 100MB per
second. Started at 6.2 GB (which is normal) and now at 20 GB and constantly
increasing.
Any other details I can provide you?
…On Wed, Jun 3, 2020 at 5:48 PM Sean McManus ***@***.***> wrote:
The normally expected behavior is that on startup, one cpptools-srv will
be launched for the active file, and then as you switch to other open
documents, additional cpptools-srv should spawned for each new TU (.cpp
file) up to your logical processor count, afterwhich the oldest used
cpptools-srv should be closed -- do you see that happening or does the
cpptools-srv count keep increasing endlessly?
Using the "Tag Parser" should fix the memory issue, but it's not
recommended since the features you use may not work (i.e. only global
symbols will be available with no TU-based info).
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#5602 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABOLYVPLUK2TKKE473HSHCTRU3VNTANCNFSM4NQ7HHZA>
.
|
So I reverted back to 0.28.0 and reloaded.
The cpptools process spawned by 0.28.2 did not get killed and was still
resident in memory. It stopped growing though, but was still holding onto
its memory (~25 GB, which is when I decided to stop). I had to manually
kill it.
…On Thu, Jun 4, 2020 at 9:46 AM Shammi Mohamed ***@***.***> wrote:
So I installed 0.28.2 today and restarted.
I notice 4 cpptools-srv processes open each taking up max of 200 MB
However cpptools process is the culprit. I just reloaded VSCode (in order
to reactivate updated cpptools extension) and since then its been
constantly chewing up memory. I can see it increase around 100MB per
second. Started at 6.2 GB (which is normal) and now at 20 GB and constantly
increasing.
Any other details I can provide you?
On Wed, Jun 3, 2020 at 5:48 PM Sean McManus ***@***.***>
wrote:
> The normally expected behavior is that on startup, one cpptools-srv will
> be launched for the active file, and then as you switch to other open
> documents, additional cpptools-srv should spawned for each new TU (.cpp
> file) up to your logical processor count, afterwhich the oldest used
> cpptools-srv should be closed -- do you see that happening or does the
> cpptools-srv count keep increasing endlessly?
>
> Using the "Tag Parser" should fix the memory issue, but it's not
> recommended since the features you use may not work (i.e. only global
> symbols will be available with no TU-based info).
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <#5602 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABOLYVPLUK2TKKE473HSHCTRU3VNTANCNFSM4NQ7HHZA>
> .
>
|
More info on the bug.. It's the cpptools process (not cpptools-srv) that ends up using excessive RAM. After launch I noticed it continuously consume more memory (around 100MB/sec). It starts at around 6.2 GB of usage and then constantly keeps increasing at that rate. Let me know if I can turn on any logging and attach specific log files. |
Is cpptools also using CPU? Does the CPU usage ever end? If you set C_Cpp.loggingLevel to "Debug" does it show any output? |
It uses nearly all of my 32 GiB of memory. |
@letmaik Do you have any more repro details? We would like to fix that, but need more info and/or a repro. |
It looks like exclude filters (files.exclude and search.exclude) are skipped when building the cache or parsing the files. I have a build tree that contains a huge temp folder (~80K files, ~30GB data). It contains mainly c/c++ files and build outputs. This folder is excluded by the filters and not visible in the explorer. cpptools uses around 16GB when this folder is part of the workspace. Doesn't matter if it is filtered or not. On the other hand when I delete this folder everything is OK. cpptools only use 50MB. |
@seyfullahari We do check files.exclude and skip parsing files under that, but we only work with glob patterns that are folders unless the C_Cpp.exclusionPolicy is changed (for performance reasons). Can you make sure your glob pattern matches to a folder and not files under the folder? If you set your C_Cpp.loggingLevel to "Debug" you can see more info on what files we are tag parsing (to see if the excluded ones are being parsed or not). There is also a hidden logging level of "9" that will output info on the exclusions that are being processed, but that level is highly verbose and not recommended to be used often. It's also possible there is something special about your scenario that is breaking our exclusion mechanism, i.e. if symlinks are used, we may not match the paths correctly for exclusion purposes -- see #4206. If you can provide more repro info maybe we can figure out the cause. |
My workspace is located at /home/.../.../projectX When I enable loglevel 9, I see thousands of logs for files located below the tmp folder. It is very obvious that tmp is being traversed. I don't see any parsing or regex match related logs. These logs are available for files that don't match the exclude filters. After 5 min of running cpptools hits ~20GB resident mem size. |
@seyfullahari I'm not able to repro the bug (and we haven't received other bug reports like this). You should see logging like:
My best guess is that a symlink exists that targets the a location under tmp. If you could provide a sample repro project that could help diagnose what is going wrong. You should still see files under tmp in the logs due to issue #3123 (i.e. our code that iterates over the files in a directory doesn't check the files.exclude until later, causing all the excluded files to be added to a vector), but that issue shouldn't cause the files to be added to the database, i.e. you shouldn't see |
cpptools uses around 2% CPU consistently.
What are you looking for in the debug log?
…On Thu, Jun 4, 2020 at 3:52 PM Sean McManus ***@***.***> wrote:
Is cpptools also using CPU? Does the CPU usage ever end? If you set
C_Cpp.loggingLevel to "Debug" does it show any output?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#5602 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABOLYVKC772HV2K522XMXQ3RVAQRNANCNFSM4NQ7HHZA>
.
|
Good news. 0.28.3 seems to be a lot more stable with regards to usage of
runaway memory usage for cpptools.
Memory usage seems to be stable at around 7.7 GB right now and holding
there as I keep opening new or switch to differnt .cpp file editors. There
seems to be a constant 2% usage. Rarely I see it go to 3%.
Earlier (0.28.1 and 0.28.2), the same workspace and repro steps would cause
a constant increase of around 100MB/sec for cpptools.
Not sure what your team did, but it looks like the issue is fixed in 0.28.3
…On Mon, Jun 15, 2020 at 2:16 PM Shammi Mohamed ***@***.***> wrote:
cpptools uses around 2% CPU consistently.
What are you looking for in the debug log?
On Thu, Jun 4, 2020 at 3:52 PM Sean McManus ***@***.***>
wrote:
> Is cpptools also using CPU? Does the CPU usage ever end? If you set
> C_Cpp.loggingLevel to "Debug" does it show any output?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <#5602 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABOLYVKC772HV2K522XMXQ3RVAQRNANCNFSM4NQ7HHZA>
> .
>
|
@shammabeth FYI, the binary cpptools is identical with 0.28.1 to 0.28.3 (and the TypeScript changes are unlikely to impact the behavior). |
Alright I can confirm for sure that in my case 0.28.3 fixes the issue. I downgraded to 0.28.2 and noticed the following
I reupgraded 0.28.3 and in comparison noticed the following
I had logging set to debug, and did not notice much difference between the 2 sessions. I did not know what I was looking for specifically in the log messages. So, it's likely that I'm missing something with that observation, but for the most the log messages seemed similar. |
@gaodingpan Yeah, we need a repro or some diagnostic logging that shows the call stack which is allocating the memory. The output of C/C++: Log Diagnostics may also help. Also, if anything can be changed to avoid the memory usage that could help isolate which system is causing the problem, such as if you add:
and do a Reset IntelliSense Database command from the Command Palette. |
Similar issue in current version (1.49.2) on MacOS 10.14.6. cpptools continuously allocates memory until system freezes (14GB). It also uses ~200% CPU when idle. |
Hey @sean-mcmanus, this issue might need further attention. @shammabeth, you can help us out by closing this issue if the problem no longer exists, or adding more information. |
Still seeing this on version 1.2.0 (Jan 14th release) I should note that I am using the Remote Development extension as well and cpptools is running on the remote server. |
@JohnRThomas We haven't been able to find a repro, so a fix would not be expected with 1.2.0-insiders2. If anyone can attach a debugger while memory is increasing fast and get a call stack that could help isolate what code is causing the issue: see https://github.com/microsoft/vscode-cpptools/wiki/Attaching-debugger-to-cpptools-or-cpptools%E2%80%90srv . |
|
@JohnRThomas Those call stacks don't show any work being done that could be accumulating memory -- looks like it's shutting down or something. Usually, the call stack that is doing "real work" and accumulating memory would be deeper than 15 frames. Are you sure you attached to the cpptools process that was using lots of CPUs? Also, you may want to use https://github.com/microsoft/vscode-cpptools/releases/tag/1.2.0-insiders2 due a fix for a bug that could cause cpptools to exit after the debugger was attached on Mac (as well as other bug fixes). |
This issue has been closed automatically because it needs more information and has not had recent activity. |
Type: General
Version 0.28.1 of cpptools uses excessive memory (56+ GB!!!!).
I am opening a folder that has 5000+ files. My system has 64GB or system RAM and a 1TB swap file on a regular disk.
I switched back to 0.28.0 and memory usage is back to normal (3.3 GB)
Describe the bug
Version 0.28.1 of cpptools uses excessive memory (56+ GB!!!!).
I work in a folder tree that has 5000+ files . My system has 64GB or system RAM and a 1TB swap file on a regular disk. I noticed system perf degradation caused by excessive use of swap partition. System monitor revealed cpptools using around 56GB. I've tried killing, restarting, rebooting the system.
Eventually I switched back to 0.28.0 of cpptools and memory usage is back to normal (3.3 GB)
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Memory usage for cpptools should be less than 5GB
Screenshots
Additional context
Downgrading to 0.28.0 fixes this issue. So it seems like something got introduced in 0.28.1 that's causing this.
The text was updated successfully, but these errors were encountered: