Reduce splitstore memory usage during chain walks

[vyzo]

Walking the chain was using a cid.Set, which can grow quite big (30M+ objects, maybe more) leading to signficant transient memory usage.

This adds an ObjectVisitor interface to use instead of a blanket cid.Set. This allows us to use a noop set during compaciton (as we already have the markset to lean on, which will be on-disk with MarkSetType = "badger") and markset-backed ones during warmup/check.

We also reset the walked block set on every iteration, to avoid building up a big one with all the blocks; we can do that because the walk is BFS.

[vyzo]

Opening as draft until I have tested timing/memory effects in my nodes.

[vyzo]

warmup with on-disk visitor is some 2x slower, takes about an hour now.

[vyzo]

marking is somewhat slower also; but on the bright side, there was no watchdog action whatsoever (which was a usual occurence).

[vyzo]

Marked as ready for review, as node burn was successful.

[vyzo]

Unified the markset/visitor dichotomy, which opens up interesting possibilities: we can now check+mark atomically using Visit.

Follow up: parallelize walkChain; yes we can now!

[vyzo]

rebasd on master.

[Stebalien]

The general logic looks good, but I feel like we can simplify a bit.

[Stebalien]

We could always copy the version, then unlock, then do the rest of this function to simplify the locking a bit.

[vyzo]

we have to hold the lock while doing the fast checks on the maps.

[Stebalien]

So, with a background worker, we can get rid of this and just have a single "writing" and "pending" set. When the background worker is done flushing the last set, it would check to see if it should flush the next one (e.g., use a channel as a flag).

That should help simplify things a bit.

[vyzo]

The disadvantage of the background worker is that we can do only one write at a time -- this can do multiple concurrent writes, which is strictly better i think.

[vyzo]

It also loses the timely error return property.

I guess we could have a worker that spawns goroutines to do writes, and somehow bubble up the error to be checked in the next operation, but this gets more complicated than I'd like it to be.

[Stebalien]

I think badger will handle write parallelism under the covers.

[vyzo]

hrm not sure -- I prefer to be explicitly parallel and also keep the ability to return an error at some writer so that we can abort.

On the same time, I don't feel terribly strongly about this and we could use a background worker who spawns writers and bubbles up the error in some way.

[vyzo]

Did a number of simplifications:

• The MarkSetVisitor interface was widened to be the sum of ObjectVisitor and MakrSet, thus obviating the need for a cast in splitstore code.
• Removed redundant writers state variable from badger markset.
• Simplified and deduplicated the code for Has and Visit by introducing tryPending and tryDB helpers.
• Added ".tmp" suffix to all transient dbs in the markset directory.

[vyzo]

pushed forgotten commit.

[Stebalien]

I don't feel too strongly either. I'll take a look at the new code.

[Stebalien]

So, most of this patch looks fine. But the locking is still a bit tricker than I'd like. I can live with everything except taking the lock in one function and dropping it in another. That's really the only blocking change.

[Stebalien]

Two non-blocking nits.

[vyzo]

rebased on master for merge, as it was a bit behind.