multilevel-down

[juliangruber]

Similar to how subleveldown fixes some issues sublevel has (not judging whether it's better in total), let's discuss whether implementing multilevel as a backend would be more useful.

Some things to discuss:

• streams: we need to expose iterators, probably with implicit buffering
• batches: we'd get the chained batch for free since that's in levelup
• encodings: we'd get encodings for free since those are in levelup
• deferred open: that's also already handled in levelup
• bundle size: when bundling for the browser, could shipping the full levelup plus read streams implementations be a problem?
• maintenance
• performance in general
• etc...

What do you think?

From the top of my head: @dominictarr @hij1nx @ralphtheninja @substack @mafintosh @maxogden @feross @jcrugzz

[ralphtheninja]

@juliangruber N00b comment incoming. I'm not familiar with "how subleveldown fixes some issues". Anyone care to fill me in briefly or link to some issues involving this?

[dominictarr]

to be honest, this seems sort of strange (subleveldown too) maybe this is a good idea... or maybe it is a good pattern. we are applying leveldown in quite a different way to what it was originally intended. It's not just a backend anymore... what is it?

[mafintosh]

@dominictarr i agree with you that it might seem a bit change at first but i like the implement-as-leveldown pattern. some pros:

• you can trust that the levelup (created with your custom leveldown) behaves exactly the same as a regular levelup.
• implementing a leveldown is quite simple using abstract-leveldown. you don't really need to worry about encodings / deferred open / callback overloads etc (see https://github.com/mafintosh/subleveldown/blob/master/leveldown.js as an example).
• there is a really good test suite for leveldowns in abstract-leveldown so you get a lot of testing for free

basically you just end up using leveldown as a "core leveldb" implementation instead of as a backend.

@ralphtheninja in dat we've hit some weird edgecases with sublevels since they don't always behave exactly as levelups, fx dominictarr/level-sublevel#78 and https://github.com/rvagg/level-spaces/issues/3 which is why we started experimenting with implementing stuff as leveldowns (fx https://github.com/mafintosh/levelup-defaults and https://github.com/mafintosh/changesdown)

[jcrugzz]

This sounds really interesting. It feels like implementing a proper leveldown in this way creates a very nice encapsulation layer to keep the client interface simple. I'd be very interested to see what this could look especially with both client and server.

[dominictarr]

Yeah, it seems like leveldown is really the core primitive not the backend. I think the biggest question for me, is how does leveldown relate to encodings? because that is the part that really differs between multilevel and leveldown.

Handling this would be the key thing to make level-sublevel be a leveldown...

Basically, it sounds like we are talking about making leveldown composable.

[juliangruber]

What do you mean by that's what differs between multilevel and leveldown? Leveldown expects strings or buffers, and multilevel can send strings and buffers

[juliangruber]

Also, if multilevel were a leveldown, could we just reuse the sublevel logic without kind of hard coding it into multilevel?

[dominictarr]

@jcrugzz there has also been talk of allowing leveldowns to handle encoding, so that when you have a leveldown that natively supports a highlevel encoding (i.e. typewise) you can do that.
This would also work really well for sublevel. Another interesting case is @substack's level-party I think it would also benefit from leveldowning multilevel

[dominictarr]

but this begs the question: what do we need levelup for? encoding and read streams?

[juliangruber]

@dominictarr that's why i'm thinking we might end up having those features in two libs, like

var db = encode(leveldown('./db'));
readStream(db).pipe(...)

[juliangruber]

hm the only thing i'm not really sure about yet is whether monkeypatching leveldown is a good idea...

[mafintosh]

@juliangruber Why do you need to monkeypatch leveldown?

[juliangruber]

To add encodings to get and put I think

[mafintosh]

couldn't the levelup just take care of that?

[juliangruber]

@mafintosh i was thinking of whether we'd even need levelup, just leveldown with wrappers

[jcrugzz]

hmm interesting, levelup still feels like the right layer to handle the encodings (which are plugged in via codec) and any surface area inputs from users, ie: the minor validation that is done. Even if leveldown can handle higher level encodings like typewise, idk if you'd really want to use it directly.. Something feels wrong about it.

[dominictarr]

@jcrugzz encoding doesn't always happen near the users. for example, in sublevel, each db adds a prefix (an array of strings) to the key, and then passes it to a core unit, which applies encodings and passes to leveldown. It helps a lot to not encode anything immediately, because then you still have access to it for pre/post hooks.

I'm wondering, maybe we can have an abstract-leveldown-wrapper which takes a leveldown, returns a new leveldown, and provides pre and post methods which are called before a leveldown method is called, or after leveldown calls back.

pre/post would allow you to change the argument, so you could apply an encoding, or transform the key... maybe pre/post could be async too?

[juliangruber]

@dominictarr exactly what i was thinking!

[mafintosh]

I had a plane ride today and took a stab at implementing multilevel as a leveldown (and using protobuf). The repo is mafintosh/multileveldown. The manifest stuff from multilevel is still missing since, at first iteration, it didn't seem to fit the leveldown pattern super well but everything else seems to work quite well and was pretty easy to implement. On the plus side encodings seems to work great since everything is just binary internally now.
I'd be happy send a massive PR with the changes if we still wanna go with the leveldown design for multilevel? :) /cc @juliangruber @dominictarr

[juliangruber]

<3! I'm for keeping it separate until we discover in real world use cases which approach works better. My gut tells me though that it's gonna be multileveldown. I'm looking at that repo :)