Simulations get super slow and consume 100% cpu

[Corstiaan84]

Hi,

Im on node v8.0.0.

When starting a simulation it's starts very swiftly and runs through the first batch of periods quickly. However, soon after it starts grinding to a very slow pace of about a period per second while consuming 100% cpu.

It also depends on the used strategy. E.g. macd runs smoothly but srsi_macd is dead slow.

Any ideas?

[Corstiaan84]

After digging around in the indicator and strategy code I think this slow perf has to do with the lookback and/or/in the for loops of the indicator?

[mikitymike]

I also have this issue when simulating for more than 40 days, regardless of period (between 100 seconds and 30m) with a custom strategy and even with noop strategy. Changing keep_lookback_periods does not seem to affect this.

If I simulate for 60 days and let it run slowly, after about 10 minutes the program crashes and it says it cannot allocate any more memory:

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

[simclem77]

same for me

[basedalexander]

In my case I can't run simulation of MACD strategy on gdax.BTC-USD for more than 60 days, then it goes very slowly, about 1 day period a minute.

@DeviaVir any thoughts why is that?

[leozzo]

add memory option to your command-line or insert it in zenbot.sh file Inviato da iPhone

…

Il giorno 22 ott 2018, alle ore 12:17, Alex Kozlov ***@***.***> ha scritto: In my case I can't run simulation of MACD strategy on gdax.BTC-USD for more than 60 days, then it goes very slowly, about 1 day period a minute. @DeviaVir any thoughts why is that? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

-- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.

[basedalexander]

@leozzo The problem is not the memory, my CPU get's 100% load from the process.

[Partiolainen]

Seems on sim command lookback and trades arrays are not reduced with any keep_lookback_periods option

I'm not spec, but something like this in sim.js will help:

function updatePeriod(trade) {
    s.period.high = Math.max(trade.price, s.period.high)
    s.period.low = Math.min(trade.price, s.period.low)
    s.period.close = trade.price
    s.period.volume += trade.size
    s.period.latest_trade_time = trade.time
    s.strategy.calculate(s)
    s.vol_since_last_blink += trade.size
    if (s.trades && s.last_trade_id !== trade.trade_id) {
      s.trades.push(trade)
      s.last_trade_id = trade.trade_id
    }
    //--- memory leak fix ---
    if(s.lookback.length > 1000){
      s.lookback.splice(-1,1)
    }

    if(s.trades.length > 1000){
      s.trades.splice(-1,1)
    }
  }

[mmdiego]

I was trying to run a long simulation (more than a month) of my custom strategy on gdax.BTC-USD at 1m period and I faced this same problem, after 15 days the simulation turned very slow and cpu went to 100%.
And then tried the fix above and it worked perfectly (which by the way must be done in engine.js, not sim.js), however limiting the size of s.lookback and s.trades affects the final simulation graph (which for me was not a problem)
So, it seems the size of these two arrays are a problem for long simulations, and I think it will be a problem as well for long live trading.
Is it neccessary to keep all the past information? Can this be implemented as a permanent fix? How can it be done for simulations for the final chart to be complete?

[mmdiego]

Reading issue #440, I saw that live trading limits the amount of data through the keep_lookback_periods param, so I answed myself that in live trading it shouldn't be a problem.
Maybe a similar parameter can be added for simulation, or some option to avoid the final graph in the sim's results so that data can be discarded for long simulations?

[mmdiego]

It seems s.trades is not useful at all for simulations.
It's only used in exitSim() in:
options_output.simresults.last_assest_value = s.trades[s.trades.length-1].price
which I think can be replaced by:
options_output.simresults.last_assest_value = s.period.close
and save a lot of memory in sims.
To get rid of s.trades, it can be done with:
if (so.mode !== 'sim' && conf.output.api.on) {

What do you think?

[mmdiego]

I've tried running the two month sim again without the size limitation on s.lookback and it worked well, so the speed problem was mainly caused by s.trades.
The fix I've proposed should improve sim performance a lot. I'll PR the changes.