Prebuilt IndexedDB format

[nolanlawson]

Shipping prebuilt SQLite databases is a common pattern in native/hybrid apps (e.g. see this, this, this, this). Currently this can be "done" on the web, but you essentially have to build up an entire IDB database on first load (slow) and then invalidate/mutate it when the underlying data store changes (slow, error-prone).

What if instead we had a standard file format for an IndexedDB database, which could be quickly loaded into the browser and accessed in a read-only way? It could be a single file, which would be fetched and cached like any other resource. All processing to convert to the underlying LevelDB/SQLite/ESE format could be done on a background thread.

Use cases:

• emoji library (most web emoji libraries run in-memory, despite the large number of emoji)
• full-text search index for a card game app
• schedule database for a conference app

It's debatable whether the database should be read-only or not, since I've seen cases where a prebuilt database is just used as a starting point for the app (e.g. a Pokédex app where the user can favorite each Pokémon), but for a v1, just keeping it read-only may be very useful. Potentially this could even encourage websites to use less memory (e.g. for the emoji use case).

[inexorabletash]

There was a related suggestion for a JSON-described schema format, which it sounds like a subset of this. You'd basically want a JSON format that describes the schema and the content of each object store.

Would JSON be sufficient? It obviously leaves out many types (e.g. dates, and anything which is serializable but not JSONable).

And then there's the question: do we think browsers would be significantly more efficient than just a small JS library that does this in a worker.

[nolanlawson]

Would JSON be sufficient?

JSON or ND-JSON might not be sufficient due to the Date issue... lots of folks are storing Dates in IDB. JSON would be very convenient for generating the files, though. Can we extend JSON with something like Date(1510258332834)?

Do we think browsers would be significantly more efficient than just a small JS library that does this in a worker

Absolutely yes, mostly due to the need to do a put() for each individual object, although perhaps that's an argument for putAll() or similar. 😃

[kaizhu256]

why can't you use JSON for date? isn't "2017-11-09T20:12:12.834Z" sufficiently precise for new Date(1510258332834).toISOString()?

[inexorabletash]

Re: Dates - you need out-of-band information to know to parse that string as a Date, which means you're not simply providing e.g. an array of values to directly put into a store. At which point, you're inventing a new serialization scheme layered on JSON. Which is fine, but it's no longer "just JSON".

[kaizhu256]

different perspective. i tend to design javascript-systems on assumption it will interact with "dumb" non-locale aware systems (sqlite3, shell-scripts, etc...). and JSON-data like {"utc_datetime":"2019-03-29T05:43:00.000Z","timezoneOffset":300} is usually sufficient.

[dmurph]

TPAC 2019 Web Apps Indexed DB triage notes:

It seems like the largest issue here is the batching of putting the data into the database. putAll should fix this issues. This is Issue #69

I believe the next issue here is the overhead of parsing the json, then turning that json into structured cloning, vs going straight from the json blob to structured cloning.

But anyways, something like issue #69 seems like the biggest win here. Maybe there would eventually be a way to stream stuff into putAll? Or give it an async iterator? Unclear.

[nolanlawson]

Given that putAll() is no longer in development (#69 (comment)), I like the idea of directly streaming JSON into an object store. As mentioned before, this won't handle Dates, but in retrospect I think this may be an edge case. My hunch is that most folks are storing objects in IndexedDB that are fully serializable as JSON. (I have no data though. 😀)

Something like this?

const reader = (await (await fetch('./array.json')).body).getReader()
const writer = db.getWriter('my_object_store') // WritableStream
await reader.pipeTo(writer)

db.getWriter() could assume a 'readwrite' transaction, and it could take an options bag like db.transaction():

const writer = db.getWriter('my_object_store', { durability: 'relaxed' })

Working with the existing db.transaction() is also possible, but it may be weird to get the microtask timing right, given that we are reading data from the network. Also making it more high-level gives more flexibility to the engine to optimize stuff (maybe?).

If we wanted to support Dates in the future, we could also always have an optional transform or something:

const writer = db.getWriter('my_object_store', { transform: item => {
  // transform a single item from the JSON array
  item.date = new Date(item.date) // parse string to Date
  return item
}})

(Maybe something like module blocks could even make it so that this transform function can run off-main-thread.)

[jimmywarting]

I think it would be bad to base this on json when indexeddb can store so much more than json types. So importing/exprting databases would be problematic for dates, blob/files, typedarrays, BigInts, Infinity, circular refs, and everything else that can be strucutred structural cloned.

cbor would be a good alternative. as it can handle all of those things.
it's a binary format. so it would be smaller in size and probably faster to decode/encode then json?