Avoid duplicate writes in Batch

[ethanfrey]

Make a new batch type that optimizes on write.

It is often that multiple operations may write to the same key (eg. deduct fee from account X, move coin from account X). We want to allow the code to be independent, but do not want to write twice to the underlying merkle tree (which is expensive).

We can add a DedupBatch, that on write will first remove all duplicates, leaving only the last set/delete that effects a given key. It can scan from the last operation, and copy operations into a clean buffer. If the key was not yet seen, add to clean buffer from the end. Then slice the clean buffer to size.

[alpe]

Sorry, hit the wrong button and accidentally closed this issue.

[ruseinov]

It sounds like a nice optimisation, but I'm not sure what this entails code-wise yet.
I'm imagining it might turn out quite hectic, because we have so many different things we could be writing at the same time, but really hard for me to estimate before I start touching this code.

But, if we don't have anything more urgent to do - it's going to be interesting to tackle.

On the other hand, logically, when we have a batch transaction which has a cost of sum of all the transactions - the i/o expenses have already been paid for :)

[ethanfrey]

This was a note-to-self over a year ago.
I'd tackle it in a day if I had time to focus on the orm layer