Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Different Heap behavior between 5.0 and 6.0. Maybe memory leak? #62182

Closed
lenew opened this issue Nov 30, 2021 · 7 comments
Closed

Different Heap behavior between 5.0 and 6.0. Maybe memory leak? #62182

lenew opened this issue Nov 30, 2021 · 7 comments
Labels
area-GC-coreclr
Milestone
7.0.0

Comments

@lenew
Copy link

lenew commented Nov 30, 2021

After upgrade my website from 5.0 to 6.0, memory consumption is about 180% as before.
Is this normal?

dotnet-counters on 5.0

[System.Runtime]
    % Time in GC since last GC (%)                                 0    
    Allocation Rate (B / 1 sec)                               49,432    
    CPU Usage (%)                                                  0    
    Exception Count (Count / 1 sec)                                0    
    GC Fragmentation (%)                                           1.23 
    GC Heap Size (MB)                                             30    
    Gen 0 GC Count (Count / 1 sec)                                 0    
    Gen 0 Size (B)                                                24    
    Gen 1 GC Count (Count / 1 sec)                                 0    
    Gen 1 Size (B)                                         1,293,576    
    Gen 2 GC Count (Count / 1 sec)                                 0    
    Gen 2 Size (B)                                        15,028,960    
    IL Bytes Jitted (B)                                    1,524,780    
    LOH Size (B)                                          10,806,240    
    Monitor Lock Contention Count (Count / 1 sec)                  0    
    Number of Active Timers                                       23    
    Number of Assemblies Loaded                                  144    
    Number of Methods Jitted                                  21,915    
    POH (Pinned Object Heap) Size (B)                        375,464    
    ThreadPool Completed Work Item Count (Count / 1 sec)           1    
    ThreadPool Queue Length                                        0    
    ThreadPool Thread Count                                        4    
    Working Set (MB)                                             240

dotnet-counters on 6.0

[System.Runtime]
    % Time in GC since last GC (%)                                 0    
    Allocation Rate (B / 1 sec)                                8,168    
    CPU Usage (%)                                                  0    
    Exception Count (Count / 1 sec)                                0    
    GC Committed Bytes (MB)                                      295    
    GC Fragmentation (%)                                           1.172
    GC Heap Size (MB)                                             88    
    Gen 0 GC Count (Count / 1 sec)                                 0    
    Gen 0 Size (B)                                                48    
    Gen 1 GC Count (Count / 1 sec)                                 0    
    Gen 1 Size (B)                                        25,512,768    
    Gen 2 GC Count (Count / 1 sec)                                 0    
    Gen 2 Size (B)                                        10,452,680    
    IL Bytes Jitted (B)                                    1,395,898    
    LOH Size (B)                                           7,419,152    
    Monitor Lock Contention Count (Count / 1 sec)                  0    
    Number of Active Timers                                       16    
    Number of Assemblies Loaded                                  141    
    Number of Methods Jitted                                  20,317    
    POH (Pinned Object Heap) Size (B)                        260,872    
    ThreadPool Completed Work Item Count (Count / 1 sec)           2    
    ThreadPool Queue Length                                        0    
    ThreadPool Thread Count                                        6    
    Time spent in JIT (ms / 1 sec)                                 0    
    Working Set (MB)                                             436
@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Nov 30, 2021
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@hez2010
Copy link
Contributor

hez2010 commented Nov 30, 2021 •
edited
Loading

We can't tell the actual problem without a gc root dump. You can try using dotnet-collect to collect a dump and analyze the gc heap following https://docs.microsoft.com/en-us/dotnet/core/diagnostics/debug-memory-leak (See Generate memory dump and Analyze the core dump sections).

@lenew
Copy link
Author

lenew commented Nov 30, 2021

Already tried that. I spent a lot time on that but get nothing, due to the enormous amount of objects. Any other method ?
Here is some eeheap on 5.0 and 6.0. 6.0 has 2 heaps and 'Commited Size' is much bigger.

Number of GC Heaps: 1
generation 0 starts at 0x00007F6AE9275F78
generation 1 starts at 0x00007F6AE8CB62E8
generation 2 starts at 0x00007F6AE7FFF000
ephemeral segment allocation context: none
         segment             begin         allocated         committed    allocated size    committed size
00007F6AE7FFE000  00007F6AE7FFF000  00007F6AEA0637F8  00007F6AEA9CD000  0x20647f8(33966072)  0x29ce000(43835392)
Large object heap starts at 0x00007F6AF7FFF000
         segment             begin         allocated         committed    allocated size    committed size
00007F6AF7FFE000  00007F6AF7FFF000  00007F6AF8982F50  00007F6AF8983000  0x983f50(9977680)  0x984000(9977856)
Pinned object heap starts at 0x00007F6AFFFFF000
00007F6AFFFFE000  00007F6AFFFFF000  00007F6B0005AAA8  00007F6B00063000  0x5baa8(375464)  0x64000(409600)
Total Allocated Size:              Size: 0x2a441f0 (44319216) bytes.
Total Committed Size:              Size: 0x3352000 (53813248) bytes.
------------------------------
GC Allocated Heap Size:    Size: 0x2a441f0 (44319216) bytes.
GC Committed Heap Size:    Size: 0x3352000 (53813248) bytes.

Number of GC Heaps: 2
------------------------------
Heap 0 (0000564D12C7BDD0)
generation 0 starts at 0x00007FAAA041DB28
generation 1 starts at 0x00007FAAA028DFA8
generation 2 starts at 0x00007FAA9FFFF000
ephemeral segment allocation context: none
         segment             begin         allocated         committed    allocated size    committed size
00007FAA9FFFE000  00007FAA9FFFF000  00007FAAA4AE4D38  00007FAAA7B30000  0x4ae5d38(78535992)  0x7b31000(129175552)
Large object heap starts at 0x00007FAC9FFFF000
         segment             begin         allocated         committed    allocated size    committed size
00007FAC9FFFE000  00007FAC9FFFF000  00007FACA03949E8  00007FACA0395000  0x3959e8(3758568)  0x396000(3760128)
Pinned object heap starts at 0x00007FACBFFFF000
00007FACBFFFE000  00007FACBFFFF000  00007FACC000D060  00007FACC0010000  0xe060(57440)  0x11000(69632)
Allocated Heap Size:       Size: 0x4e89780 (82352000) bytes.
Committed Heap Size:       Size: 0x7ec7000 (132935680) bytes.
------------------------------
Heap 1 (0000564D12C82060)
generation 0 starts at 0x00007FABA09D0A18
generation 1 starts at 0x00007FABA08D2888
generation 2 starts at 0x00007FAB9FFFF000
ephemeral segment allocation context: none
         segment             begin         allocated         committed    allocated size    committed size
00007FAB9FFFE000  00007FAB9FFFF000  00007FABA4DCFC20  00007FABA817D000  0x4dd0c20(81595424)  0x817e000(135782400)
Large object heap starts at 0x00007FACAFFFF000
         segment             begin         allocated         committed    allocated size    committed size
00007FACAFFFE000  00007FACAFFFF000  00007FACB055B088  00007FACB055C000  0x55c088(5619848)  0x55d000(5623808)
Pinned object heap starts at 0x00007FACCFFFF000
00007FACCFFFE000  00007FACCFFFF000  00007FACD0030AA8  00007FACD0040000  0x31aa8(203432)  0x41000(266240)
Allocated Heap Size:       Size: 0x535e750 (87418704) bytes.
Committed Heap Size:       Size: 0x86db000 (141406208) bytes.
------------------------------
GC Allocated Heap Size:    Size: 0xa1e7ed0 (169770704) bytes.
GC Committed Heap Size:    Size: 0x105a2000 (274341888) bytes.

@ghost
Copy link

ghost commented Nov 30, 2021

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

After upgrade my website from 5.0 to 6.0, memory consumption is about 180% as before.
Is this normal?

dotnet-counters on 5.0

[System.Runtime]
    % Time in GC since last GC (%)                                 0    
    Allocation Rate (B / 1 sec)                               49,432    
    CPU Usage (%)                                                  0    
    Exception Count (Count / 1 sec)                                0    
    GC Fragmentation (%)                                           1.23 
    GC Heap Size (MB)                                             30    
    Gen 0 GC Count (Count / 1 sec)                                 0    
    Gen 0 Size (B)                                                24    
    Gen 1 GC Count (Count / 1 sec)                                 0    
    Gen 1 Size (B)                                         1,293,576    
    Gen 2 GC Count (Count / 1 sec)                                 0    
    Gen 2 Size (B)                                        15,028,960    
    IL Bytes Jitted (B)                                    1,524,780    
    LOH Size (B)                                          10,806,240    
    Monitor Lock Contention Count (Count / 1 sec)                  0    
    Number of Active Timers                                       23    
    Number of Assemblies Loaded                                  144    
    Number of Methods Jitted                                  21,915    
    POH (Pinned Object Heap) Size (B)                        375,464    
    ThreadPool Completed Work Item Count (Count / 1 sec)           1    
    ThreadPool Queue Length                                        0    
    ThreadPool Thread Count                                        4    
    Working Set (MB)                                             240

dotnet-counters on 6.0

[System.Runtime]
    % Time in GC since last GC (%)                                 0    
    Allocation Rate (B / 1 sec)                                8,168    
    CPU Usage (%)                                                  0    
    Exception Count (Count / 1 sec)                                0    
    GC Committed Bytes (MB)                                      295    
    GC Fragmentation (%)                                           1.172
    GC Heap Size (MB)                                             88    
    Gen 0 GC Count (Count / 1 sec)                                 0    
    Gen 0 Size (B)                                                48    
    Gen 1 GC Count (Count / 1 sec)                                 0    
    Gen 1 Size (B)                                        25,512,768    
    Gen 2 GC Count (Count / 1 sec)                                 0    
    Gen 2 Size (B)                                        10,452,680    
    IL Bytes Jitted (B)                                    1,395,898    
    LOH Size (B)                                           7,419,152    
    Monitor Lock Contention Count (Count / 1 sec)                  0    
    Number of Active Timers                                       16    
    Number of Assemblies Loaded                                  141    
    Number of Methods Jitted                                  20,317    
    POH (Pinned Object Heap) Size (B)                        260,872    
    ThreadPool Completed Work Item Count (Count / 1 sec)           2    
    ThreadPool Queue Length                                        0    
    ThreadPool Thread Count                                        6    
    Time spent in JIT (ms / 1 sec)                                 0    
    Working Set (MB)                                             436
Author: lenew
Assignees: -
Labels:

area-GC-coreclr, untriaged
Milestone: -

@mangod9
Copy link
Member

mangod9 commented Nov 30, 2021

is your app using workstation or server gc? Since the number of heaps has changed (assuming without any code/config changes on your side) wonder if you are hitting: #61674.

@mangod9 mangod9 removed the untriaged New issue has not been triaged by the area owner label Nov 30, 2021
@mangod9 mangod9 added this to the 7.0.0 milestone Nov 30, 2021
@lenew
Copy link
Author

lenew commented Dec 1, 2021

Yes, workstation mode in container. The same problem in #61674.
Mem usage back to normal after change "System.GC.Server": false in runtimesconfig.json.

@lenew lenew closed this as completed Dec 1, 2021
@mangod9
Copy link
Member

mangod9 commented Dec 1, 2021

Thanks @lenew for validating, we hope to backport fix for #61674 to 6.0.

@mangod9 mangod9 mentioned this issue Dec 2, 2021
Merged
@ghost ghost locked as resolved and limited conversation to collaborators Dec 31, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-GC-coreclr
Projects
None yet
Milestone
7.0.0
Development

No branches or pull requests
4 participants
@lenew @huoyaoyuan @hez2010 @mangod9