From 65b2315702c40cb2ceb8effcb14a1a43547accdb Mon Sep 17 00:00:00 2001
From: pubkey <8926560+pubkey@users.noreply.github.com>
Date: Fri, 10 May 2024 21:30:54 +0200
Subject: [PATCH 01/15] ADD article
---
.../localstorage-indexeddb-cookies-opfs-sqlite-wasm.md | 10 ++++++++++
1 file changed, 10 insertions(+)
create mode 100644 docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
diff --git a/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
new file mode 100644
index 00000000000..eb672740240
--- /dev/null
+++ b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
@@ -0,0 +1,10 @@
+# Localstorage vs. IndexedDB vs. Cookies vs. OPFS vs. Wasm-SQLite
+
+
+
+### Things this does not talk about
+WebSQL
+session storage.
+Web Storage API
+
+
From 501718a64eb3e137e43fbc25c1b6eba2f00dfd83 Mon Sep 17 00:00:00 2001
From: pubkey <8926560+pubkey@users.noreply.github.com>
Date: Tue, 14 May 2024 01:20:53 +0200
Subject: [PATCH 02/15] ADD content
---
...rage-indexeddb-cookies-opfs-sqlite-wasm.md | 92 ++++++++++++++++++-
docs-src/docs/articles/localstorage.md | 2 +-
2 files changed, 92 insertions(+), 2 deletions(-)
diff --git a/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
index eb672740240..2d9207aa28b 100644
--- a/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
+++ b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
@@ -1,10 +1,100 @@
# Localstorage vs. IndexedDB vs. Cookies vs. OPFS vs. Wasm-SQLite
+So it is 2024 and you want to build your this awesome Web Application. To make you app fast and even work [offline](../offline-first.md), you want to store all data on the client device and run operations there, instead of awaiting requests to a backend server.
+But there is a problem:
-### Things this does not talk about
+> Browser are slow and weren't made to run huge database operations.
+
+Or are they? Over the last few years, a lot of features have been added to JavaScript. New storage APIs such as OPFS and new features like the BroadcastChannel are here to let us reach the limits of JavaScript performance.
+
+In this article we have look at all previous and new technologies to store and query data in a browser. We will run performance tests and combine many performance hacks to find out how fast we can write and read a **huge amount of data** in a Web App.
+
+
+## What is localstorage
+## What is indexeddb
+## What are cookies
+## What is OPFS
+## What is wasm sqlite
+
+## Things this does not talk about
WebSQL
session storage.
Web Storage API
+
+
+## Feature comparison
+
+### Multitab support
+
+A big difference when building a Web App compared to Electron or React-Native, is that the user will open and close the app in multiple browser tabs at the same time. Therefore you have not only one JavaScript process running, but many of them can exist and might have to share state changes between each other to not show outdated data to the user.
+
+Not all storage APIs support a way to automatically share write events between tabs. Only localstorage has the [storage-event](./localstorage.md#localstorage-vs-indexeddb) which can be used to observe changes.
+
+```js
+// localStorage can observe changes with the storage event.
+// This feature is missing in IndexedDB and others
+addEventListener("storage", (event) => {});
+```
+
+To workaround this problem, there are two solutions:
+The first option is to use the [BroadcastChannel API](https://github.com/pubkey/broadcast-channel) which can send messages across browser tabs. So whenever you do a write to the storage, you also send a notification to other tabs to inform them about these changes.
+
+The other solution is to use the [SharedWorker](https://developer.mozilla.org/en-US/docs/Web/API/SharedWorker) and do all writes inside of the worker. All browser tabs can then subscribe to messages from that SharedWorker and know about changes.
+
+### Indexing
+- IndexedDB does not support boolean indexes
+#### iterable indexes
+#### secondary indexes
+### Storing complex JSON documents
+
+### Worker Support
+
+- Localstorage and Cookies [cannot be used in WebWorker or SharedWorker](https://stackoverflow.com/questions/6179159/accessing-localstorage-from-a-webworker).
+
+- OPFS with the fast `createSyncAccessHandle` method can **only** [be used in a WebWorker](../rx-storage-opfs.md#opfs-limitations).
+
+## Performance Comparison
+
+Now that we know about the basics of the various storage solutions, lets compare their performance. As mentioned above, we not only want to store a few documents, instead lets store **many** of them and run **heavy and complex queries**.
+
+TODO add github repo url with performance tests.
+
+### Running many small operations
+
+One aspect of performance is the latency. The time to run a small database operation, either read or write.
+Depending on your use case, it might be relevant that many small operations run fast, like when you have a browser game and want to store the game's state.
+
+### Running single big operations
+
+### Initial page load
+How fast does the first query load when there are many documents
+stored already.
+
+
+
+
+## Lets reach the limits of client side storage performance with RxDB
+- indexeddb optimizations
+- compression with keycompression
+- spliting work load with WebWorker
+- sharding
+- memory mapped/synced stuff
+
+- Store metadata in localstorage
+- fix initial page load for new tabs with the SharedWorker (only chrome can spawn WebWorkers inside of a SharedWorker)
+
+- is OPFS faster then indexeddb?
+
+
+
+## Read further
+
+TODO fix links
+- Check out the [hackernews discussion of this article](https://news.ycombinator.com/item?id=39745993)
+- Shared/Like my [announcement tweet](https://twitter.com/rxdbjs/status/1769507055298064818)
+
+- Learn how to use RxDB with the [RxDB Quickstart](../quickstart.md)
+- Check out the [RxDB github repo](https://github.com/pubkey/rxdb) and leave a star ⭐
diff --git a/docs-src/docs/articles/localstorage.md b/docs-src/docs/articles/localstorage.md
index aa5552020f3..b5910a73b7b 100644
--- a/docs-src/docs/articles/localstorage.md
+++ b/docs-src/docs/articles/localstorage.md
@@ -92,7 +92,7 @@ While **localStorage** serves as a reliable storage solution for simpler data ne
complex queries can pose a challenge with IndexedDB, and while its performance is acceptable, IndexedDB can be [too slow](../slow-indexeddb.md) for some use cases.
```js
-// localStorage can observe chanes with the storage event.
+// localStorage can observe changes with the storage event.
// This feature is missing in IndexedDB
addEventListener("storage", (event) => {});
```
From 5386bbd8ccb7bf8fa0683ee9c4bdc7a655fbcaf8 Mon Sep 17 00:00:00 2001
From: pubkey <8926560+pubkey@users.noreply.github.com>
Date: Wed, 15 May 2024 14:23:44 +0200
Subject: [PATCH 03/15] chore
---
...alstorage-indexeddb-cookies-opfs-sqlite-wasm.md | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
index 6946baf747a..162d0ccab5a 100644
--- a/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
+++ b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
@@ -47,17 +47,27 @@ The first option is to use the [BroadcastChannel API](https://github.com/pubkey/
The other solution is to use the [SharedWorker](https://developer.mozilla.org/en-US/docs/Web/API/SharedWorker) and do all writes inside of the worker. All browser tabs can then subscribe to messages from that SharedWorker and know about changes.
### Indexing
-- IndexedDB does not support boolean indexes
+
+The big difference between a database and storing data in a plain file, is that a database is writing data in a format that allows running operations over indexes to facilitate fast queries.
+
#### iterable indexes
+- IndexedDB does not support boolean indexes
+
#### secondary indexes
+- Only IndexedDB and SQLite WASM has secondary indexes
+
+
### Storing complex JSON documents
+- IndexedDB can store JSON natively
+- SQLite can [store JSON](https://www.sqlite.org/json1.html) from version 3.38.0 (2022-02-22).
+
### Worker Support
- Localstorage and Cookies [cannot be used in WebWorker or SharedWorker](https://stackoverflow.com/questions/6179159/accessing-localstorage-from-a-webworker).
- OPFS with the fast `createSyncAccessHandle` method can **only** [be used in a WebWorker](../rx-storage-opfs.md#opfs-limitations).
-
+ 2
## Performance Comparison
Now that we know about the basics of the various storage solutions, lets compare their performance. As mentioned above, we not only want to store a few documents, instead lets store **many** of them and run **heavy and complex queries**.
From c1647ea1ce965ef58625bf3ead77feb5cbcefcc0 Mon Sep 17 00:00:00 2001
From: pubkey <8926560+pubkey@users.noreply.github.com>
Date: Sat, 8 Jun 2024 22:39:34 +0200
Subject: [PATCH 04/15] CHORE content
---
...rage-indexeddb-cookies-opfs-sqlite-wasm.md | 31 ++++++++++++++-----
1 file changed, 24 insertions(+), 7 deletions(-)
diff --git a/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
index 162d0ccab5a..ff47dbe2b20 100644
--- a/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
+++ b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
@@ -1,22 +1,39 @@
# Localstorage vs. IndexedDB vs. Cookies vs. OPFS vs. Wasm-SQLite
-So it is 2024 and you want to build your this awesome Web Application. To make you app fast and even work [offline](../offline-first.md), you want to store all data on the client device and run operations there, instead of awaiting requests to a backend server.
-But there is a problem:
+Welcome to 2024, a time when building powerful, responsive web applications has never been more exciting. As developers, we constantly seek ways to make our apps faster and more efficient, and one crucial aspect of this is data storage. To ensure our apps not only perform well but also work seamlessly [offline](../offline-first.md), we aim to store data on the client device, minimizing the need to interact with backend servers.
-> Browser are slow and weren't made to run huge database operations.
+However, a common belief persists:
-Or are they? Over the last few years, a lot of features have been added to JavaScript. New storage APIs such as OPFS and new features like the BroadcastChannel are here to let us reach the limits of JavaScript performance.
+> Browser are slow and weren't designed to run extensive database operations.
-In this article we have look at all previous and new technologies to store and query data in a browser. We will run performance tests and combine many performance hacks to find out how fast we can write and read a **huge amount of data** in a Web App.
+Is this really the case? Over recent years, JavaScript has evolved significantly. New storage APIs like the Origin Private File System (OPFS) and new features like the BroadcastChannel have pushed the boundaries of what JavaScript can achieve in terms of performance.
+In this article, we will delve into the various technologies available for storing and querying data in a browser. We'll explore traditional methods like Cookies, LocalStorage and IndexedDB, and newer solutions such as OPFS and Wasm-SQLite. Through performance tests we aim to uncover how fast we can write and read **a huge amount of data** in a web application with the various methods. And because you are reading this in the [RxDB](/) docs, we will utilize multiple RxDB plugins that contain innovative hacks to reach the performance limits of a browser in terms of database operations.
+
+
+
## What is localstorage
-## What is indexeddb
+
+LocalStorage provides a simple way to store key-value pairs in a web browser. It's suitable for storing small amounts of data that need to persist across sessions but is [limited by a 5MB storage cap](./localstorage.md#understanding-the-limitations-of-local-storage) and the inability to store complex data types beyond strings.
+
## What are cookies
+
+Cookies store small pieces of data that are sent with every HTTP request. They are mainly used for session management, personalization, and tracking, but are limited to about `4 KB` of data in [RFC-6265](https://datatracker.ietf.org/doc/html/rfc6265#section-6.1).
+So we cannot store much data in a cookie but it is still interesting how good cookie access performance compared to the other methods. Especially because cookies are such an important base feature of the web, many performance optimizations have been done and even these days there is still progress being made like the [Shared Memory Versioning](https://blog.chromium.org/2024/06/introducing-shared-memory-versioning-to.html) by chromium.
+
+## What is IndexedDB
+
+
## What is OPFS
-## What is wasm sqlite
+## What is wasm sqlite
+https://www.fermyon.com/blog/webassembly-wasi-and-the-component-model
### Things this does not talk about
From a9a596ca5fa0893c348d016219ddee1bda7e8b37 Mon Sep 17 00:00:00 2001
From: pubkey <8926560+pubkey@users.noreply.github.com>
Date: Tue, 11 Jun 2024 02:26:32 +0200
Subject: [PATCH 05/15] CHORE text
---
...rage-indexeddb-cookies-opfs-sqlite-wasm.md | 24 ++++++++++++++-----
1 file changed, 18 insertions(+), 6 deletions(-)
diff --git a/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
index ff47dbe2b20..3584919339b 100644
--- a/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
+++ b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
@@ -10,7 +10,7 @@ However, a common belief persists:
Is this really the case? Over recent years, JavaScript has evolved significantly. New storage APIs like the Origin Private File System (OPFS) and new features like the BroadcastChannel have pushed the boundaries of what JavaScript can achieve in terms of performance.
-In this article, we will delve into the various technologies available for storing and querying data in a browser. We'll explore traditional methods like Cookies, LocalStorage and IndexedDB, and newer solutions such as OPFS and Wasm-SQLite. Through performance tests we aim to uncover how fast we can write and read **a huge amount of data** in a web application with the various methods. And because you are reading this in the [RxDB](/) docs, we will utilize multiple RxDB plugins that contain innovative hacks to reach the performance limits of a browser in terms of database operations.
+In this article, we will dive into the various technologies available for storing and querying data in a browser. We'll explore traditional methods like Cookies, LocalStorage and IndexedDB, and newer solutions such as OPFS and Wasm-SQLite. Through performance tests we aim to uncover how fast we can write and read **a huge amount of data** in a web application with the various methods. And because you are reading this in the [RxDB](/) docs, we will utilize multiple RxDB plugins that contain innovative hacks to reach the performance limits of a browser in terms of database operations.
-## What is localstorage
+## What is Localstorage
LocalStorage provides a simple way to store key-value pairs in a web browser. It's suitable for storing small amounts of data that need to persist across sessions but is [limited by a 5MB storage cap](./localstorage.md#understanding-the-limitations-of-local-storage) and the inability to store complex data types beyond strings.
-## What are cookies
+## What are Cookies
Cookies store small pieces of data that are sent with every HTTP request. They are mainly used for session management, personalization, and tracking, but are limited to about `4 KB` of data in [RFC-6265](https://datatracker.ietf.org/doc/html/rfc6265#section-6.1).
-So we cannot store much data in a cookie but it is still interesting how good cookie access performance compared to the other methods. Especially because cookies are such an important base feature of the web, many performance optimizations have been done and even these days there is still progress being made like the [Shared Memory Versioning](https://blog.chromium.org/2024/06/introducing-shared-memory-versioning-to.html) by chromium.
+This means we cannot store much data in a cookie but it is still interesting how good cookie access performance compared to the other methods. Especially because cookies are such an important base feature of the web, many performance optimizations have been done and even these days there is still progress being made like the [Shared Memory Versioning](https://blog.chromium.org/2024/06/introducing-shared-memory-versioning-to.html) by chromium.
## What is IndexedDB
+IndexedDB is a low-level API for storing large amounts of structured (JSON) data. While the API is a bit hard to use, IndexedDB can utilize indexes and asynchronous operations. It lacks support for complex queries and only allows to iterate over the indexes which makes it more like a base layer for other libraries. The performance of basic IndexedDB operations can be problematic but there exist [several hacks](../slow-indexeddb.md) to improve writes and query speed.
+
## What is OPFS
+The [Origin Private File System](../rx-storage-opfs.md) (OPFS) is a relatively new API that allows web applications to store large files directly in the browser. It is designed for data-intensive applications that want to write and read binary data.
+OPFS can be used in two modes: Either asynchronous on the [main thread](../rx-storage-opfs.md#using-opfs-in-the-main-thread-instead-of-a-worker) or in a WebWorker with the faster, aynchronous access.
+Because only binary data can be processed, OPFS is made to be as a base filesystem for database libraries. You will unlikely directly want to use the OPFS in your applications code.
+
+
+## What is WASM-SQLite
+
+SQLite is a small, fast, self-contained SQL database written in the C programming language.
+Because browsers cannot run an applications C code directly, [WebAssembly](https://webassembly.org/) (WASM) is used to compile the SQLite C code into WASM byte code. WASM code can be shipped to browser apps and generally runs much faster compared to JavaScript, but still about [10% slower then native](https://www.usenix.org/conference/atc19/presentation/jangda).
+The compiled byte code has a size of [about 938.9 kB](https://sqlite.org/download.html) which must be downloaded and parsed by the users on the first page load.
+
+WASM cannot directly access any persistend storage API in the browser. Instead it requires data to flow from WASM to the main-thread and then can be put into one of the browser APIs. For reads the same goes the other way round.
-## What is wasm sqlite
-https://www.fermyon.com/blog/webassembly-wasi-and-the-component-model
### Things this does not talk about
From be4128204fe6321bb9ce8b4b417804d95c8face4 Mon Sep 17 00:00:00 2001
From: pubkey <8926560+pubkey@users.noreply.github.com>
Date: Mon, 17 Jun 2024 12:51:33 +0200
Subject: [PATCH 06/15] CHORe
---
...rage-indexeddb-cookies-opfs-sqlite-wasm.md | 50 +++++++++++--------
1 file changed, 29 insertions(+), 21 deletions(-)
diff --git a/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
index 3584919339b..0be96d1417c 100644
--- a/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
+++ b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
@@ -20,7 +20,7 @@ In this article, we will dive into the various technologies available for storin
## What is Localstorage
-LocalStorage provides a simple way to store key-value pairs in a web browser. It's suitable for storing small amounts of data that need to persist across sessions but is [limited by a 5MB storage cap](./localstorage.md#understanding-the-limitations-of-local-storage) and the inability to store complex data types beyond strings.
+LocalStorage provides a simple API to store key-value pairs in a web browser. It's suitable for storing small amounts of data that need to persist across sessions but is [limited by a 5MB storage cap](./localstorage.md#understanding-the-limitations-of-local-storage) and the inability to store complex data types beyond strings.
## What are Cookies
@@ -47,11 +47,23 @@ The compiled byte code has a size of [about 938.9 kB](https://sqlite.org/downlo
WASM cannot directly access any persistend storage API in the browser. Instead it requires data to flow from WASM to the main-thread and then can be put into one of the browser APIs. For reads the same goes the other way round.
+## Test Setup
-### Things this does not talk about
-WebSQL
-session storage.
-Web Storage API
+As mentioned above, we will focus on the performance differences of the various technologies.
+But we not only want to store a few documents, instead lets store **many** of them and run **heavy and complex queries** to find out about the limits of what can be done in a browser.
+
+TODO add github repo url with performance tests. https://github.com/pubkey/localstorage-indexeddb-cookies-opfs-sqlite-wasm
+
+### Running many small operations
+
+One aspect of performance is the latency. The time to run a small database operation, either read or write.
+Depending on your use case, it might be relevant that many small operations run fast, like when you have a browser game and want to store the game's state.
+
+### Running single big operations
+
+### Initial page load
+How fast does the first query load when there are many documents
+stored already.
@@ -97,22 +109,6 @@ The big difference between a database and storing data in a plain file, is that
- OPFS with the fast `createSyncAccessHandle` method can **only** [be used in a WebWorker](../rx-storage-opfs.md#opfs-limitations).
2
-## Performance Comparison
-
-Now that we know about the basics of the various storage solutions, lets compare their performance. As mentioned above, we not only want to store a few documents, instead lets store **many** of them and run **heavy and complex queries**.
-
-TODO add github repo url with performance tests.
-
-### Running many small operations
-
-One aspect of performance is the latency. The time to run a small database operation, either read or write.
-Depending on your use case, it might be relevant that many small operations run fast, like when you have a browser game and want to store the game's state.
-
-### Running single big operations
-
-### Initial page load
-How fast does the first query load when there are many documents
-stored already.
@@ -129,6 +125,15 @@ stored already.
- is OPFS faster then indexeddb?
+### Things this article does not talk about
+
+WebSQL
+session storage.
+Web Storage API
+Cross tab support
+Observability
+
+
## Read further
@@ -136,6 +141,9 @@ stored already.
TODO fix links
- Check out the [hackernews discussion of this article](https://news.ycombinator.com/item?id=39745993)
- Shared/Like my [announcement tweet](https://twitter.com/rxdbjs/status/1769507055298064818)
+- Reproduce the benchmarks at the [github repo](https://github.com/pubkey/localstorage-indexeddb-cookies-opfs-sqlite-wasm)
- Learn how to use RxDB with the [RxDB Quickstart](../quickstart.md)
- Check out the [RxDB github repo](https://github.com/pubkey/rxdb) and leave a star ⭐
+
+
From 10cbc9f4c160c752529323f3de468b4ef901d932 Mon Sep 17 00:00:00 2001
From: pubkey <8926560+pubkey@users.noreply.github.com>
Date: Wed, 26 Jun 2024 12:11:04 +0200
Subject: [PATCH 07/15] CHORE text
---
...alstorage-indexeddb-cookies-opfs-sqlite-wasm.md | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)
diff --git a/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
index 0be96d1417c..e2472c74368 100644
--- a/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
+++ b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
@@ -10,7 +10,7 @@ However, a common belief persists:
Is this really the case? Over recent years, JavaScript has evolved significantly. New storage APIs like the Origin Private File System (OPFS) and new features like the BroadcastChannel have pushed the boundaries of what JavaScript can achieve in terms of performance.
-In this article, we will dive into the various technologies available for storing and querying data in a browser. We'll explore traditional methods like Cookies, LocalStorage and IndexedDB, and newer solutions such as OPFS and Wasm-SQLite. Through performance tests we aim to uncover how fast we can write and read **a huge amount of data** in a web application with the various methods. And because you are reading this in the [RxDB](/) docs, we will utilize multiple RxDB plugins that contain innovative hacks to reach the performance limits of a browser in terms of database operations.
+In this article, we will dive into the various technologies available for storing and querying data in a browser. We'll explore traditional methods like Cookies, LocalStorage and IndexedDB, and newer solutions such as OPFS and Wasm-SQLite. Through performance tests we aim to uncover how fast we can write and read **a huge amount of data** in a web application with the various methods. And because you are reading this in the [RxDB](/) docs, we will utilize multiple RxDB plugins that contain innovative concepts to reach the performance limits of a browser in terms of database operations.
+:::
+
+
+## The available storage APIs in a modern browser
+
+Over the years, the type of web applications we build has changed significantly. In the early years of the web we served static html files. Then we served dynamically rendered html and later we build single page applications that run most logic on the client. And for the comming years you might want to build so called [local first apps](../offline-first.md) that handle big and complex data operations solely on the client and even work when offline which gives you the opportunity to build zero-latency user interactions.
+
+For these increments of use cases, the browser vendors kept in pace providing more and better APIs. Let me give you a brief explanation of them:
+
+### What are Cookies
+
+Cookies were first introduced by [netscape in 1994](https://www.baekdal.com/thoughts/the-original-cookie-specification-from-1997-was-gdpr-compliant/).
+Cookies store small pieces of key-value data. Cookies are mainly used for session management, personalization, and tracking, but are limited to about `4 KB` of data in [RFC-6265](https://datatracker.ietf.org/doc/html/rfc6265#section-6.1). Because the stored cookies are send to the server with every HTTP request, this limitation is reasonable. You can test your browsers cookie limits [here](http://www.ruslog.com/tools/cookies.html).
-## What is Localstorage
+This size limitation means we cannot store much data in a cookie but it is still interesting how good cookie access performance compared to the other methods. Especially because cookies are such an important base feature of the web, many performance optimizations have been done and even these days there is still progress being made like the [Shared Memory Versioning](https://blog.chromium.org/2024/06/introducing-shared-memory-versioning-to.html) by chromium or the asynchronous [CookieStore API](https://developer.mozilla.org/en-US/docs/Web/API/Cookie_Store_API).
-LocalStorage provides a simple API to store key-value pairs in a web browser. It's suitable for storing small amounts of data that need to persist across sessions but is [limited by a 5MB storage cap](./localstorage.md#understanding-the-limitations-of-local-storage) and the inability to store complex data types beyond strings.
-## What are Cookies
+### What is Localstorage
-Cookies store small pieces of data that are sent with every HTTP request. They are mainly used for session management, personalization, and tracking, but are limited to about `4 KB` of data in [RFC-6265](https://datatracker.ietf.org/doc/html/rfc6265#section-6.1). You can test your browsers cookie limits [here](http://www.ruslog.com/tools/cookies.html).
+The [LocalStorage API](./localstorage.md) was first proposed as part of the [WebStorage specification in 2009](https://www.w3.org/TR/2009/WD-webstorage-20090423/#the-localstorage-attribute).
+LocalStorage provides a simple API to store key-value pairs inside of a web browser. It has the methods `setItem`, `getItem`, `removeItem` and `clear` which is all you need from a key-value store. Localstorage is only suitable for storing small amounts of data that need to persist across sessions and it is [limited by a 5MB storage cap](./localstorage.md#understanding-the-limitations-of-local-storage). Storing complex data is only possible by transforming it into a string for example with `JSON.stringify()`.
+The API is not asynchronous which means if fully blocks your JavaScript process while doing stuff. Therefore running heavy operations on it might block your UI from rendering.
-This limitations means we cannot store much data in a cookie but it is still interesting how good cookie access performance compared to the other methods. Especially because cookies are such an important base feature of the web, many performance optimizations have been done and even these days there is still progress being made like the [Shared Memory Versioning](https://blog.chromium.org/2024/06/introducing-shared-memory-versioning-to.html) by chromium.
+> There is also the SesssionStorage API. The key difference is that localStorage data persists indefinitely until explicitly cleared, while sessionStorage data is cleared when the browser tab or window is closed
-## What is IndexedDB
+
+
+
+-----------------------------------------------------------------------------
+-----------------------------------------------------------------------------
+-----------------------------------------------------------------------------
+-----------------------------------------------------------------------------
+
+### What is IndexedDB
IndexedDB is a low-level API for storing large amounts of structured (JSON) data. While the API is a bit hard to use, IndexedDB can utilize indexes and asynchronous operations. It lacks support for complex queries and only allows to iterate over the indexes which makes it more like a base layer for other libraries. The performance of basic IndexedDB operations can be problematic but there exist [several hacks](../slow-indexeddb.md) to improve writes and query speed.
-## What is OPFS
+### What is OPFS
The [Origin Private File System](../rx-storage-opfs.md) (OPFS) is a relatively new API that allows web applications to store large files directly in the browser. It is designed for data-intensive applications that want to write and read binary data.
OPFS can be used in two modes: Either asynchronous on the [main thread](../rx-storage-opfs.md#using-opfs-in-the-main-thread-instead-of-a-worker) or in a WebWorker with the faster, aynchronous access.
Because only binary data can be processed, OPFS is made to be as a base filesystem for database libraries. You will unlikely directly want to use the OPFS in your applications code.
-## What is WASM-SQLite
+### What is WASM-SQLite
SQLite is a small, fast, self-contained SQL database written in the C programming language.
Because browsers cannot run an applications C code directly, [WebAssembly](https://webassembly.org/) (WASM) is used to compile the SQLite C code into WASM byte code. WASM code can be shipped to browser apps and generally runs much faster compared to JavaScript, but still about [10% slower then native](https://www.usenix.org/conference/atc19/presentation/jangda).
@@ -48,6 +94,11 @@ The compiled byte code has a size of [about 938.9 kB](https://sqlite.org/downlo
WASM cannot directly access any persistend storage API in the browser. Instead it requires data to flow from WASM to the main-thread and then can be put into one of the browser APIs. For reads the same goes the other way round.
+### What is WebSQL
+
+
+
+
## Test Setup
As mentioned above, we will focus on the performance differences of the various technologies.
diff --git a/docs-src/sidebars.js b/docs-src/sidebars.js
index a6b9000616a..6cdafd2e86c 100644
--- a/docs-src/sidebars.js
+++ b/docs-src/sidebars.js
@@ -153,6 +153,7 @@ const sidebars = {
'articles/ionic-database',
'articles/json-database',
'articles/websockets-sse-polling-webrtc-webtransport',
+ 'articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm',
'articles/localstorage',
'articles/mobile-database',
'articles/progressive-web-app-database',
From 37e9aa80d8f81db13d319ef1ee02fbe6054407a9 Mon Sep 17 00:00:00 2001
From: pubkey <8926560+pubkey@users.noreply.github.com>
Date: Wed, 9 Oct 2024 10:11:47 +0200
Subject: [PATCH 09/15] CHORE
---
docs-src/docs/articles/ideas.md | 2 +-
...rage-indexeddb-cookies-opfs-sqlite-wasm.md | 40 +++++++++++--------
src/rx-collection-helper.ts | 1 -
3 files changed, 25 insertions(+), 18 deletions(-)
diff --git a/docs-src/docs/articles/ideas.md b/docs-src/docs/articles/ideas.md
index 0483c6f430e..adaacf8435b 100644
--- a/docs-src/docs/articles/ideas.md
+++ b/docs-src/docs/articles/ideas.md
@@ -5,7 +5,7 @@
- Finding the optimal way to shorten vector embeddings
- Performance and quality of vector comparison functions (euclideanDistance etc)
- performance and quality of vector indexing methods
-
+- What is new in IndexedDB 3.0
## Seo keywords:
diff --git a/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
index ea40a2cd5d9..9b6f3ead673 100644
--- a/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
+++ b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
@@ -64,39 +64,47 @@ The [LocalStorage API](./localstorage.md) was first proposed as part of the [Web
LocalStorage provides a simple API to store key-value pairs inside of a web browser. It has the methods `setItem`, `getItem`, `removeItem` and `clear` which is all you need from a key-value store. Localstorage is only suitable for storing small amounts of data that need to persist across sessions and it is [limited by a 5MB storage cap](./localstorage.md#understanding-the-limitations-of-local-storage). Storing complex data is only possible by transforming it into a string for example with `JSON.stringify()`.
The API is not asynchronous which means if fully blocks your JavaScript process while doing stuff. Therefore running heavy operations on it might block your UI from rendering.
-> There is also the SesssionStorage API. The key difference is that localStorage data persists indefinitely until explicitly cleared, while sessionStorage data is cleared when the browser tab or window is closed
+> There is also the **SessionStorage** API. The key difference is that localStorage data persists indefinitely until explicitly cleared, while sessionStorage data is cleared when the browser tab or window is closed.
+### What is IndexedDB
+IndexedDB was first introduced as "Indexed Database API" [in 2015](https://www.w3.org/TR/IndexedDB/#sotd).
------------------------------------------------------------------------------
------------------------------------------------------------------------------
------------------------------------------------------------------------------
------------------------------------------------------------------------------
+IndexedDB is a low-level API for storing large amounts of structured JSON data. While the API is a bit hard to use, IndexedDB can utilize indexes and asynchronous operations. It lacks support for complex queries and only allows to iterate over the indexes which makes it more like a base layer for other libraries then a fully fledged database.
-### What is IndexedDB
+In 2018, IndexedDB version 2.0 [was introduced](https://hacks.mozilla.org/2016/10/whats-new-in-indexeddb-2-0/). This added some major improvements. Most noticeable the `getAll()` method which improves performance dramatically when fetching bulks of JSON documents.
-IndexedDB is a low-level API for storing large amounts of structured (JSON) data. While the API is a bit hard to use, IndexedDB can utilize indexes and asynchronous operations. It lacks support for complex queries and only allows to iterate over the indexes which makes it more like a base layer for other libraries. The performance of basic IndexedDB operations can be problematic but there exist [several hacks](../slow-indexeddb.md) to improve writes and query speed.
+IndexedDB [version 3.0](https://w3c.github.io/IndexedDB/) is in the workings which contains many improvements. Most important the addition of `Promise` based calls that makes modern JS features like `async/await` more useful.
### What is OPFS
-The [Origin Private File System](../rx-storage-opfs.md) (OPFS) is a relatively new API that allows web applications to store large files directly in the browser. It is designed for data-intensive applications that want to write and read binary data.
-OPFS can be used in two modes: Either asynchronous on the [main thread](../rx-storage-opfs.md#using-opfs-in-the-main-thread-instead-of-a-worker) or in a WebWorker with the faster, aynchronous access.
-Because only binary data can be processed, OPFS is made to be as a base filesystem for database libraries. You will unlikely directly want to use the OPFS in your applications code.
+The [Origin Private File System](../rx-storage-opfs.md) (OPFS) is a [relatively new](https://caniuse.com/mdn-api_filesystemfilehandle_createsyncaccesshandle) API that allows web applications to store large files directly in the browser. It is designed for data-intensive applications that want to write and read **binary data** in a simulated file system.
-### What is WASM-SQLite
+OPFS can be used in two modes:
+- Either asynchronous on the [main thread](../rx-storage-opfs.md#using-opfs-in-the-main-thread-instead-of-a-worker)
+- Or in a WebWorker with the faster, aynchronous access with the `createSyncAccessHandle()` method.
-SQLite is a small, fast, self-contained SQL database written in the C programming language.
-Because browsers cannot run an applications C code directly, [WebAssembly](https://webassembly.org/) (WASM) is used to compile the SQLite C code into WASM byte code. WASM code can be shipped to browser apps and generally runs much faster compared to JavaScript, but still about [10% slower then native](https://www.usenix.org/conference/atc19/presentation/jangda).
-The compiled byte code has a size of [about 938.9 kB](https://sqlite.org/download.html) which must be downloaded and parsed by the users on the first page load.
+Because only binary data can be processed, OPFS is made to be a base filesystem for library developers. You will unlikely directly want to use the OPFS in your code when you build a "normal" application because it is too complex. That would only make sense for storing plain files like images, not to store and query JSON data efficiently.
-WASM cannot directly access any persistend storage API in the browser. Instead it requires data to flow from WASM to the main-thread and then can be put into one of the browser APIs. For reads the same goes the other way round.
+### What is WASM SQLite
+[WebAssembly](https://webassembly.org/) (Wasm) is a binary format that allows high-performance code execution on the web.
+Wasm was added to major browsers over the course of 2017 wich opened a wide range of opportunities on what to run inside of a browser. You can compile native libraries to WebAssembly and just run them on the client with just a few adjustments. WASM code can be shipped to browser apps and generally runs much faster compared to JavaScript, but still about [10% slower then native](https://www.usenix.org/conference/atc19/presentation/jangda).
-### What is WebSQL
+Many people started to use compiled SQLite as a database inside of the browser which is why it makes sense to also compare this setup to the native APIs.
+
+The compiled byte code of SQLite has a size of [about 938.9 kB](https://sqlite.org/download.html) which must be downloaded and parsed by the users on the first page load. WASM cannot directly access any persistend storage API in the browser. Instead it requires data to flow from WASM to the main-thread and then can be put into one of the browser APIs. For reads the same goes the other way round.
+-----------------------------------------------------------------------------
+-----------------------------------------------------------------------------
+-----------------------------------------------------------------------------
+-----------------------------------------------------------------------------
+
+
+### What is WebSQL
## Test Setup
diff --git a/src/rx-collection-helper.ts b/src/rx-collection-helper.ts
index a52d0a3da5d..f43960a5df7 100644
--- a/src/rx-collection-helper.ts
+++ b/src/rx-collection-helper.ts
@@ -95,7 +95,6 @@ export async function removeCollectionStorages(
);
const relevantCollectionMetaDocs = allCollectionMetaDocs
.filter(metaDoc => metaDoc.data.name === collectionName);
-
let removeStorages: {
collectionName: string;
schema: RxJsonSchema;
From 92905d567debe8e242232936218fbcf002f68f26 Mon Sep 17 00:00:00 2001
From: pubkey <8926560+pubkey@users.noreply.github.com>
Date: Thu, 10 Oct 2024 10:58:02 +0200
Subject: [PATCH 10/15] CHORE
---
...rage-indexeddb-cookies-opfs-sqlite-wasm.md | 133 ++++++++++++------
1 file changed, 93 insertions(+), 40 deletions(-)
diff --git a/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
index 9b6f3ead673..c04677e6ba9 100644
--- a/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
+++ b/docs-src/docs/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md
@@ -34,7 +34,7 @@ In this article, we will dive into the various technologies available for storin
:::note
You are reading this in the [RxDB](/) docs. RxDB is a JavaScript database that has different storage adapters which can utilize the different storage APIs.
-Over the last 8 years I spend most of my time working with these APIs, doing performance tests and building [hacks](../slow-indexeddb.md) to reach the limits of browser database operation speed.
+**Since 2017** I spend most of my time working with these APIs, doing performance tests and building [hacks](../slow-indexeddb.md) to reach the limits of browser database operation speed.
+
+
+
+