Improve performance by switching from pandas to numpy

[maurerle]

During my recent studies I found ASSUME to be very slow when simulating a whole year.
One way to improve this is by switching towards daily market clearing instead of hourly, but it still takes a while.
When looking into the code which takes a lot of time I found pandas to often be the case:

Act 1: Profiling Benchmarks

due to various reasons, cProfile does not give good timings when running async code.
More correct timings can be seen using yappi (pip install yappi) - https://github.com/sumerc/yappi
So one can run yappi -o "out.profile" cli.py and then use tuna (pip install tuna) to visualize the profiling result:
tuna out.profile.
This gives theses visual charts like shown below.
The results are therefore equally to running assume -s example01a -c base. This run takes 88s on my laptop.
Probably ~20s are spent organizing asyncio-stuff
~60s is spent in pandas
~3s on imports
~ rest on other stuff

calculate_bids boils down to take time in pandas

handling market_feedback spends a lot of time in pandas too - nearly all the site-packages stuff is spend in pandas

writing outputs spends most of its time in pandas too

Though one can not see that much due to the long lines - I could not find a way to remove the absolute paths from the pictures..

Act 2: Alternatives

So I thought how one can replace pandas.
Our requirements includes slicing, indexing by datetime and having multiple series.
After experimenting with modin and dask
I could not use modin as a drop in replacement and dask did not seem like a good solution either, as we spend a lot of time in the initialization of dataframes and not in the heavy lifting.

I came up with good old numpy, which supports slicing. But can only have an array with the same types.
So a datetime index is not possible.

I thought about having a convenience wrapper - something like this:

def idx_from_date(date):
    return (date-start)//freq

def numpy_dt_indexer(data, fr, to):
    from_idx = idx_from_date(fr)
    to_idx = idx_from_date(to)
    return data[from_idx:to_idx]

After all, it turns out, that switching to numpy is at least 40x faster than pandas.
I really hope, that this is also the case when switching the main parts of the simulation to it.

Act 3

Implementation
TBD