Always crash when the memory increased to 4~5G

[mike442144]

• Version:v6.10.3
• Platform:Linux compute 4.4.0-72-generic [stream] support piping multiple streams into a single stream #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Subsystem:

I have 14G memory on my machinek, but when I run my program with following command, the memory usage increased to 4~5G then crashed.

node --max-old-space-size=10000

and here is the stack trace.

<--- Last few GCs --->

  366988 ms: Scavenge 4288.9 (4709.6) -> 4273.4 (4709.6) MB, 3.6 / 0.0 ms [allocation failure].
  367158 ms: Scavenge 4289.7 (4710.6) -> 4274.2 (4710.6) MB, 3.6 / 0.0 ms [allocation failure].
  367330 ms: Scavenge 4290.5 (4710.6) -> 4275.0 (4710.6) MB, 4.3 / 0.0 ms [allocation failure].
  370489 ms: Scavenge 4803.3 (5223.6) -> 4787.7 (5223.6) MB, 3.8 / 0.0 ms (+ 8.1 ms in 76 steps since last GC) [allocation failure].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2ca34c4cfb39 <JS Object>
    1: aggregate(aka aggregate) [/mnt/tmall/tmall-utils/lib/gmv.js:~57] [pc=0x75800b6ff8] (this=0x2ca34c404381 <undefined>,record=0x1deb1543c0a1 <an Object with map 0x16ea92811c49>)
    2: _tickCallback [internal/process/next_tick.js:~93] [pc=0x75800bc41c] (this=0xff389a89db9 <a process with map 0x16ea92811671>)
    3: /* anonymous */(aka /* anonymous */) [module.js:606] [pc=0x7580035aad] (this...

FATAL ERROR: invalid table size Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x10d309c [node]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape, v8::internal::Handle<v8::internal::Name> >::New(v8::internal::Isolate*, int, v8::internal::MinimumCapacity, v8::internal::PretenureFlag) [node]
 6: v8::internal::Dictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape, v8::internal::Handle<v8::internal::Name> >::EnsureCapacity(v8::internal::Handle<v8::internal::NameDictionary>, int, v8::internal::Handle<v8::internal::Name>) [node]
 7: v8::internal::Dictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape, v8::internal::Handle<v8::internal::Name> >::Add(v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails) [node]
 8: v8::internal::JSObject::AddSlowProperty(v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes) [node]
 9: v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::Object::StoreFromKeyed) [node]
10: v8::internal::Object::SetProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::LanguageMode, v8::internal::Object::StoreFromKeyed) [node]
11: v8::internal::Runtime::SetObjectProperty(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::LanguageMode) [node]
12: v8::internal::Runtime_SetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [node]
13: 0x757ff092a7
Aborted (core dumped)

Anybody know this issue?

[mscdex]

Looks like you probably have a leak somewhere, so you might try running the script through a memory profiler, such as the one in the node/V8 inspector.

I'm not sure that V8 supports a larger max memory usage for a single process on 64-bit.

[vsemozhetbyt]

cc @nodejs/v8

[addaleax]

According to the V8 5.8 release blogpost, unlimited heap sizes are a relatively new thing, and prior to that 4GB was the usual limit.

V8 5.8 will be the version coming with Node 8.0.0 – we’ll have the first release candidate for that later this week, if you’re interested.

[mike442144]

@mscdex A memory leak won't lead the program crash I think, just keep eating memory.
So it seems to be a limitation of v8. Am I right?
@addaleax Certainly, if possible I'm going to upgrade the node.

[vsemozhetbyt]

@mike442144 You can even try the first nightly build with v8 5.8.

[mike442144]

@vsemozhetbyt I'm trying it...

[addaleax]

@mike442144 Unless you know why your process is using that much memory, a memory leak is the most likely explanation for the process running out of memory.

[mike442144]

@addaleax Of course I know why, I'm computing about 20 million records. :)

[mscdex]

@mike442144 But all of them in memory at the same time?

[mike442144]

@mscdex @addaleax @vsemozhetbyt I have tried the nightly build of 8 which avoided out of memory. But I have another problem, when I use Map with 10 million keys, it is rather slow to call get() function. Is there any benchmark of the Map structure?

[mike442144]

@mscdex Yes, I have to compute all of them in memory, I find it is difficult to do data analysis in nodejs. Maybe python is much more suitable? Do you have any experiences of this?

[bnoordhuis]

I'd say the original question has been answered. @mike442144 Please post follow-up questions to https://github.com/nodejs/help/issues.

[mike442144]

@bnoordhuis Thanks, certainly!