javascript out of memory

[BarnumD]

I've been seeing some out of memory errors when doing backfill. I'm running in a docker container (barnumd/zenbot:3.2.4). There's no memory limit on the container. And my host has 16GB of RAM, 12 available.

`08/20/2016 05:51:56 PM UTC [ reducer] processing 9/ticks sec, thought queue: 297075
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

<--- Last few GCs --->

7925875 ms: Mark-sweep 1292.6 (1434.9) -> 1292.1 (1434.9) MB, 2954.8 / 0 ms [allocation failure] [GC in old space requested].
7928486 ms: Mark-sweep 1292.1 (1434.9) -> 1292.0 (1434.9) MB, 2611.1 / 0 ms [allocation failure] [GC in old space requested].
7930980 ms: Mark-sweep 1292.0 (1434.9) -> 1291.8 (1434.9) MB, 2494.4 / 0 ms [last resort gc].
7933455 ms: Mark-sweep 1291.8 (1434.9) -> 1291.8 (1434.9) MB, 2474.5 / 0 ms [last resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1cb8528c9e31
1: parse [native json.js:~51] [pc=0x3479c74f0a43] (this=0x1cb8528c37b1 <a JSON with map 0x3012fee095e9>,x=0x37a77b01d4a9 <String[254]: {"app":"zb","id":"zb:poloniex-43478444","key":"trade","value":{"id":"poloniex-43478444","trade_id":43478444,"time":1469526543000,"asset":"ETH","currency":"BTC","size":1.71176473,"price":0.02030962,"side":"buy","exchange":"poloniex"},"time":1469526543000}>,s=...

1: node::Abort() [node]
2: 0xfb1c8c [node]
3: v8::Utils::ReportApiFailure(char const_, char const_) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const_, bool) [node]
5: v8::internal::Factory::NewTransitionArray(int) [node]
6: v8::internal::TransitionArray::Allocate(v8::internal::Isolate_, int, int) [node]
7: v8::internal::TransitionArray::Insert(v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::Name, v8::internal::Handlev8::internal::Map, v8::internal::SimpleTransitionFlag) [node]
8: v8::internal::Map::CopyReplaceDescriptors(v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::DescriptorArray, v8::internal::Handlev8::internal::LayoutDescriptor, v8::internal::TransitionFlag, v8::internal::MaybeHandlev8::internal::Name, char const_, v8::internal::SimpleTransitionFlag) [node]
9: v8::internal::Map::CopyAddDescriptor(v8::internal::Handlev8::internal::Map, v8::internal::Descriptor_, v8::internal::TransitionFlag) [node]
10: v8::internal::Map::CopyWithField(v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::Name, v8::internal::Handlev8::internal::FieldType, v8::internal::PropertyAttributes, v8::internal::Representation, v8::internal::TransitionFlag) [node]
11: v8::internal::Map::TransitionToDataProperty(v8::internal::Handlev8::internal::Map, v8::internal::Handlev8::internal::Name, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes, v8::internal::Object::StoreFromKeyed) [node]
12: v8::internal::LookupIterator::PrepareTransitionToDataProperty(v8::internal::Handlev8::internal::JSObject, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes, v8::internal::Object::StoreFromKeyed) [node]
13: v8::internal::Object::AddDataProperty(v8::internal::LookupIterator_, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::Object::StoreFromKeyed) [node]
14: v8::internal::JSObject::DefineOwnPropertyIgnoreAttributes(v8::internal::LookupIterator_, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::JSObject::AccessorInfoHandling) [node]
15: v8::internal::JSObject::DefinePropertyOrElementIgnoreAttributes(v8::internal::Handlev8::internal::JSObject, v8::internal::Handlev8::internal::Name, v8::internal::Handlev8::internal::Object, v8::internal::PropertyAttributes) [node]
16: v8::internal::JsonParser::ParseJsonObject() [node]
17: v8::internal::JsonParser::ParseJsonValue() [node]
18: v8::internal::JsonParser::ParseJson() [node]
19: v8::internal::Runtime_ParseJson(int, v8::internal::Object*, v8::internal::Isolate) [node]
20: 0x3479c710961b
08/20/2016 05:52:40 PM UTC [ reducer] processing 4/ticks sec, thought queue: 292575
`

[grigio]

Can you link the dockerfile?

[BarnumD]

Actual run command I'm using... (The default run doesn't backfill so I pass this command the first run after removing my db for a new version.)
docker run -d -v /docker/mounts/zenbot/config.js:/usr/src/zenbot/config.js -p 3013:3013 --link mongodb:mongodb --cpu-shares=2 --name zenbot barnumd/zenbot:3.2.4 zenbot launch map --backfill reduce run server

Here's the dockerfile.
https://github.com/BarnumD/docker-zenbot/blob/master/Dockerfile

[BarnumD]

Host is Ubuntu 16.04. 1.12.0-rc4 (I should update :) )

[grigio]

I run it on Ubuntu 16.04 with node 5.9.1 and node 6.4 and I never had that kind of problem.. 2gb RAM 6gb swap. Probably your node version is buggy

[BarnumD]

I'll try node 6.4.0. Also updated to docker 1.12.1

[carlos8f]

I'm using Ubuntu 14.04 2gb ram no swap, node 6.3.1, mongo 3.2.8, reduce command when processing a large queue sometimes crashes with a mongo timeout, respawns and then clears the queue quickly.

@BarnumD from your paste it looks like 297075 items in your thought queue, which means the reduce command is way behind. Thought queue should be around 2000-8000 items during backfill. If the thought queue keeps creeping up, check your mongodb.log for slow queries, check your config.js and remove exchanges/pairs you're not using, relax the backfill timeout, until the thought queue can be efficiently processed.

Hope that helps

[carlos8f]

you can also try backfilling a single exchange or pair at a time, if your reducer can't keep up with all of them at once

[BarnumD]

OK, I do have more than the default pairs and exchanges enabled. I also saw some timeouts from the mongodb. So that sounds accurate. Is backfilling a single exchange at a time in the readme/docs?

[carlos8f]

Is backfilling a single exchange at a time in the readme/docs?

No, but I plan on updating readme/docs with conversations like these under the FAQ section as they come up.

If anyone would like to add things to the FAQ please submit Pull Requests :)

[BarnumD]

My assets were:
"BTC",
"ETH",
And currencies were:
"USD",
"USDT",
"BTC"
Exchanges:
"bitfinex",
"gdax",
"poloniex",
"server"

I removed the BTC currency and so far my thought queue is <= 100

[carlos8f]

what would be interesting is if Zenbrain (once the auto-learn engine is done) could tweak its own map/reduce parameters to adjust to the system it's running on. realtime perf analysis of its own perf :)

[grigio]

I think the config.js should only have BTC/USD by default.. also because the logic doesn't support other pairs as it is now https://github.com/carlos8f/zenbot/blob/master/default_logic.js#L45-L46

Another ambiguos part is that is that also the currencies specified in config.js could be valid for an exchange but invalid for another

[BarnumD]

I did let it run for a while with both ETH and BTC assets running and the queue did get out of hand again with very few tick/s.

Alright. I'll just separate asset into their own running instances and mongo databases.