Infinite memory usage on first run

[cuuupid]

I'm using LinvoDB3 with leveldown in pure Node, with two simple models:

Message:

{
  locations: [
    {
      folder: { type: String },
      uid: { type: Number },
    }
  ],
  mid: { type: String, index: true },
  tid: { type: String },
}

Thread:

{
  mids: [],
  tid: { type: String, index: true }
}

On the very first run, it lets me use the database, but starts gradually using up more and more memory while I perform other operations (like TLS connections). Everything slows down and then I get this heap error from Node:

--- Last few GCs --->

[28372:0x10428e000]   162828 ms: Mark-sweep (reduce) 2020.9 (2051.7) -> 2020.3 (2053.0) MB, 9311.3 / 0.0 ms  (average mu = 0.097, current mu = 0.004) task scavenge might not succeed
[28372:0x10428e000]   169765 ms: Mark-sweep (reduce) 2021.4 (2052.0) -> 2020.8 (2053.2) MB, 6885.0 / 0.0 ms  (average mu = 0.053, current mu = 0.007) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
 1: 0x100c23213 node::Abort() (.cold.1) [/usr/local/bin/node]
 2: 0x10008833d node::FatalError(char const*, char const*) [/usr/local/bin/node]
 3: 0x1000884a6 node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 4: 0x10018f2c9 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0x10018f273 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 6: 0x1002b6679 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 7: 0x1002ee4dd v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [/usr/local/bin/node]
 8: 0x1002de5d0 void v8::internal::LiveObjectVisitor::VisitBlackObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor, v8::internal::MajorNonAtomicMarkingState>(v8::internal::MemoryChunk*, v8::internal::MajorNonAtomicMarkingState*, v8::internal::EvacuateNewSpaceVisitor*, v8::internal::LiveObjectVisitor::IterationMode) [/usr/local/bin/node]
 9: 0x1002de1d6 v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [/usr/local/bin/node]
10: 0x1002ddf2b v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [/usr/local/bin/node]
11: 0x1002f1299 v8::internal::PageEvacuationTask::ProcessItems() [/usr/local/bin/node]
12: 0x1002f1234 v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [/usr/local/bin/node]
13: 0x1002c829e v8::internal::ItemParallelJob::Task::RunInternal() [/usr/local/bin/node]
14: 0x1002c8629 v8::internal::ItemParallelJob::Run() [/usr/local/bin/node]
15: 0x1002df98b void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [/usr/local/bin/node]
16: 0x1002df78f v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [/usr/local/bin/node]
17: 0x1002d3b79 v8::internal::MarkCompactCollector::Evacuate() [/usr/local/bin/node]
18: 0x1002d1a40 v8::internal::MarkCompactCollector::CollectGarbage() [/usr/local/bin/node]
19: 0x1002b6bbc v8::internal::Heap::MarkCompact() [/usr/local/bin/node]
20: 0x1002b4fb8 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
21: 0x1002b3979 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
22: 0x1002bc3ae v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
23: 0x1002bc6ce v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
24: 0x10029b467 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/local/bin/node]
25: 0x10050fdbf v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
26: 0x100793d99 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]
27: 0x100816d91 Builtins_CreateFunctionContextHandler [/usr/local/bin/node]
28: 0x10072d142 Builtins_InterpreterEntryTrampoline [/usr/local/bin/node]
Abort trap: 6

Increasing the memory allocated to the process just makes it use up more memory and increase the heap further (even when supplied 128GB memory which is insane).

However, after the first run, everything works perfectly and is totally fine.

Of course, this isn't consistent -- if I only add one document for the Message and Thread model, it initializes the DB and storage files correctly but it's not until I run the script a second time and try adding several dozen documents that it freezes up, takes up all possibly memory, and then crashes.

I have no idea what could be happening right now and would appreciate any help!

[Ivshti]

sounds like there's a infinite loop or recursion somewhere that keeps allocating objects but unfortunately i've forgotten the source code over the years

look for the indexing code and in particular loops or functions that call themselves through async callbacks

[cuuupid]

It doesn't look like the callback context is being disposed of. The culprit was a TLS connection I had made, consumed and closed in the callback of a model .save. While I would expect the relevant memory to be GC'ed after returning from the callback, instead it seems it was keeping it in memory (and so storing 1000's of TLS connections and their buffers).

To resolve it, I just dumped the result into a queue that I have constantly being consumed to form the relevant connections. Don't know why GC seems to be ignoring the callback.