Prefer backpressure over snapshot guarantees #86
Labels
Comments
|
I'm not sure whether to include this in v3 or exactly how to implement it. It can also be added in a minor later, but the thing that bothers me is that the readme states "Store large amounts of data in modern browsers". You can write large amounts of data, sure, but without backpressure, you're gonna run into trouble when reading. |
|
@ralphtheninja thoughts? |
|
I'd skip it for v3 so we can get |
Closed
|
Relevant: Level/community#63 (comment) and Level/community#63 (comment). IMO offering proper backpressure is a higher priority now. |
|
I now prefer that we forgo snapshotting guarantees, in favor of backpressure and avoiding a blocking transaction. |
vweevers
changed the title
|
Plan:
|
vweevers
added a commit
to Level/browser-level
that referenced
this issue
Mar 5, 2022
Previously (in `level-js`) an iterator would keep reading in the background so as to keep the IndexedDB transaction alive and thus not see the data of simultaneous writes. I.e. it was reading from a snapshot in time. The downsides of that approach: - Memory usage on large databases - IndexedDB doesn't actually use a snapshot (Chrome used to) but rather a blocking transaction. Meaning you can't write while an iterator is reading. So instead, an iterator now reads a few entries ahead and then opens a new transaction on the next read. A "few" means all entries for `iterator.all()`, `size` amount of entries for `iterator.nextv(size)` and a hardcoded 100 entries for `iterator.next()`. Individual calls to those methods still have snapshot guarantees, but repeated calls do not. Reading should now be faster too, because it uses the `getAll()` and `getAllKeys()` methods of IndexedDB, instead of a cursor. This means multiple entries are transferred from IndexedDB to JS in a single turn of the JS event loop, rather than one turn per entry. Reverse iterators do still use a cursor and are therefor slower. To reflect the new behavior, `db.supports.snapshots` is now false. Ref Level/level-js#86
|
Done in |
Repository owner
moved this from Todo
to Done
in Level
Mar 5, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Continuing from #68.
TBD if this should go in v3.
The text was updated successfully, but these errors were encountered: