[sqlite][ios] introduce sqlite/next #24812

Kudo · 2023-10-10T14:30:00Z

Why

close ENG-9488
close ENG-9890
close ENG-10388
close ENG-10394

the doc updates and android implementation will be in separate prs.

How

drop websql interfaces and rewrite everything apis in expo-sqlite/next.
the apis are heavily inspired by node-sqlite3 and better-sqlite3.

Test Plan

✅ ts-only unit tests through better-sqlite3
✅ e2e tests in SQLiteNext.ts

Checklist

Documentation is up to date to reflect these changes (eg: https://docs.expo.dev and README.md).
Conforms with the Documentation Writing Style Guide
This diff will work correctly for npx expo prebuild & EAS Build (eg: updated a module plugin).

linear · 2023-10-10T14:30:07Z

ENG-9488 Support prepared statements in expo-sqlite

#23728 (comment)

ENG-9890 Drop WebSQL interfaces

Originally expo-sqlite is built on top on WebSQL interface, so that we have some legacy code that is not reasonable to SQLite. For example, exec with readOnly support (in statement level) is not natural to SQLite.

Given that WebSQL is deprecated, we should just drop WebSQL support.

alanjhughes · 2023-10-11T12:57:46Z

Looks great @Kudo 👏👏 Added @tsapeta and @lukmccall to get some more opinions

Kudo · 2023-10-11T14:23:38Z

Looks great @Kudo 👏👏 Added @tsapeta and @lukmccall to get some more opinions

@tsapeta suggested me to try the shared objects from expo-modules-core. i'm looking into the integration. will update this pr later for the shared objects integration.

Kudo · 2023-10-12T16:08:53Z

updated the pr for shared objects we could go ahead to review the pr again.
i also found an issue about shared objects and would be resolved by #24836

tsapeta

Love it! 💙 Pretty powerful example of what we can build with shared objects!

I wonder if we could also have synchronous versions of these methods? Are they expensive to run?

packages/expo-sqlite/ios/Exceptions.swift

packages/expo-sqlite/ios/NativeDatabase.swift

packages/expo-sqlite/ios/NativeStatement.swift

tsapeta · 2023-10-13T17:35:02Z

packages/expo-sqlite/ios/SQLAction.swift

+  case unknown
+
+  static func fromCode(value: Int32) -> SQLAction {
+    switch value {


What are the other cases? Can we maybe link to a documentation where we can find what these "magic" numbers mean? I think it's especially important that there are no specific constants which names would already mean something.

There are 34 of them 😅 In our case, for the update hook, these are the ones we are interested in but the full list is here https://www.sqlite.org/c3ref/c_alter_table.html

tsapeta · 2023-10-13T17:52:30Z

packages/expo-sqlite/src/next/Database.ts

+   * @param source A string containing the SQL query.
+   * @param params Parameters to bind to the query.
+   */
+  public runAsync(source: string, ...params: VariadicBindParams): Promise<RunResult>;


I'm wondering if this will render correctly in our docs 🤔

looks fine. please see #24962 and preview at http://docs.expo.dev-pr-24962.s3-website-us-east-1.amazonaws.com/versions/unversioned/sdk/sqlite-next/

packages/expo-sqlite/src/next/__mocks__/ExpoSQLiteNext.ts

tsapeta · 2023-10-13T17:56:40Z

packages/expo-sqlite/src/next/NativeDatabase.ts

+/**
+ * A class that represents an instance of the SQLite database.
+ */
+export declare class NativeDatabase {


I really like that we don't need any additional JS code here, just the types 😍

# Why the weak shared objects will have exception when deallocating on hermes engine. # How - when the concrete js object is deallocated, we don't have to reset its `__expo_shared_object_id__` property. # Test Plan tested on #24812 + shared objects integration (still local, will update #24812 later)

brentvatne · 2023-10-18T22:05:46Z

packages/expo-sqlite/src/next/Database.ts

+   */
+  public async transactionAsync(txn: () => Promise<void>): Promise<void> {
+    try {
+      await this.nativeDatabase.execAsync('BEGIN');


this seems susceptible to the problem that matt brought up in notion: https://www.notion.so/CRSqlite-REST-syncing-example-69b0d4423d5547c98f24b80fe2323071?d=1f7423c8e5ce4510b26abfcac9bc0815&pvs=4#172d47cb5f794c90aeb4f69f70af4b2b -- i don't think this is a dealbreaker for this api, just something that people might want to watch out for that we may need to document.

in general, what do you think about adding support for sync functions to the api as well?

I agree with adding sync functions too, if that doesn't degrade the performance. See my question from #24812 (review)

i have been convinced by Brent on a call to support sync apis. my only concern is about the api naming. i will do in a separate pr.

we also talked about the problem of async transaction api. i'm still looking for a proper solution.

this seems susceptible to the problem that matt brought up in notion: https://www.notion.so/CRSqlite-REST-syncing-example-69b0d4423d5547c98f24b80fe2323071?d=1f7423c8e5ce4510b26abfcac9bc0815&pvs=4#172d47cb5f794c90aeb4f69f70af4b2b

addressed by c11839e from the new transactionExclusiveAsync call. close ENG-10394

Co-authored-by: Tomasz Sapeta <[email protected]>

Kudo · 2023-10-19T09:24:07Z

rebase main to fix the ci errors.

…oblem

Kudo · 2023-10-20T14:23:54Z

both sync apis and async transaction issues are resolved. the pr is good to review again. also having #24962 for doc update.

linear · 2023-10-20T14:44:48Z

ENG-10394 Enhance transactionAsync in `expo-sqlite/next`

As mentioned by Matt in the comment, async transaction may introduce some out-of-order issue, we should find a way to enhance the API to be more solid.

Asides from serialized call or locking call, there's another option to create a new connection (sqlite3_open) to the same db for transaction. Knex uses this design and they have a connection poll to support transactions. I would prefer this approach without connection pool. We should document that transactionAsync would create a new connection with some extra effort and may see SQLITE_BUSY error.

ENG-10388 Having synchronous API from `expo-sqlite/next`

It has some use cases to have synchronous API in advanced and would be good if we could support this.

However, I don't like to have similar APIs and confuse people, like closeAsync and close/closeSync. Maybe we could have something to use synchronous API series with the hint that the APIs would block JavaScript thread:

blocking_close, blocking_exec
sync_close, sync_exec
import { close, exec } from 'expo-sqlite/next/sync';

# Why follow up with #24812 to add android implementation # How - add android implementation - [js] update `addDatabaseChangeListener` listener arguments where `dbName = 'main'` by default and having `dbFilePath` to the exact opened file path - [js] update `getAsync` and `getSync` to fix android [returns undefined result](https://github.com/expo/expo/blob/7204c425c62306b7b72c01e1e065a52f97905377/packages/expo-modules-core/android/src/main/java/expo/modules/kotlin/jni/JavaCallback.kt#L16-L19) # Test Plan test-suite SQLiteNext passed on android

# Why the weak shared objects will have exception when deallocating on hermes engine. # How - when the concrete js object is deallocated, we don't have to reset its `__expo_shared_object_id__` property. # Test Plan tested on #24812 + shared objects integration (still local, will update #24812 later)

# Why close ENG-9488 close ENG-9890 close ENG-10388 close ENG-10394 the doc updates and android implementation will be in separate prs. # How drop websql interfaces and rewrite everything apis in `expo-sqlite/next`. the apis are heavily inspired by node-sqlite3 and better-sqlite3. # Test Plan - ts-only unit tests through better-sqlite3 - e2e tests in SQLiteNext.ts

# Why follow up with #24812 to add android implementation # How - add android implementation - [js] update `addDatabaseChangeListener` listener arguments where `dbName = 'main'` by default and having `dbFilePath` to the exact opened file path - [js] update `getAsync` and `getSync` to fix android [returns undefined result](https://github.com/expo/expo/blob/7204c425c62306b7b72c01e1e065a52f97905377/packages/expo-modules-core/android/src/main/java/expo/modules/kotlin/jni/JavaCallback.kt#L16-L19) # Test Plan test-suite SQLiteNext passed on android

# Why following up #24812 to update the doc --------- Co-authored-by: Aman Mittal <[email protected]> Co-authored-by: Bartosz Kaszubowski <[email protected]>

# Why following up expo#24812 to update the doc --------- Co-authored-by: Aman Mittal <[email protected]> Co-authored-by: Bartosz Kaszubowski <[email protected]>

Kudo force-pushed the @kudo/sqlite-next branch from ca24747 to 66d2188 Compare October 10, 2023 14:33

expo-bot added the bot: needs changes ExpoBot found things that don't meet our guidelines label Oct 10, 2023

github-actions bot added the bot: fingerprint changed label Oct 10, 2023

expo-bot added bot: passed checks ExpoBot has nothing to complain about and removed bot: needs changes ExpoBot found things that don't meet our guidelines labels Oct 10, 2023

Kudo force-pushed the @kudo/sqlite-next branch from 1e3f265 to 5ff2d42 Compare October 11, 2023 02:09

Kudo marked this pull request as ready for review October 11, 2023 06:37

Kudo requested a review from alanjhughes October 11, 2023 06:38

alanjhughes requested review from tsapeta and lukmccall October 11, 2023 12:54

Kudo mentioned this pull request Oct 12, 2023

[core] fix shared object exception for thread issues #24836

Merged

3 tasks

Kudo force-pushed the @kudo/sqlite-next branch from 5ff2d42 to 2679388 Compare October 12, 2023 14:13

tsapeta reviewed Oct 13, 2023

View reviewed changes

brentvatne reviewed Oct 18, 2023

View reviewed changes

Kudo and others added 6 commits October 19, 2023 17:18

[sqlite] introduce sqlite/next

183783c

update changelog

dc919d6

fix swiftlint issue

cff4f73

fix android test error

0bb2121

migrate to shared objects

af5bcdb

Apply suggestions from code review

6cd231e

Co-authored-by: Tomasz Sapeta <[email protected]>

Kudo force-pushed the @kudo/sqlite-next branch from 35ef387 to 6cd231e Compare October 19, 2023 09:24

Kudo added 2 commits October 20, 2023 04:44

Introduce transactionExclusiveAsync to deal with async transaction pr…

c11839e

…oblem

Add synchronous APIs

42c785c

Kudo mentioned this pull request Oct 20, 2023

[doc] add sqlite/next doc #24962

Merged

3 tasks

tsapeta approved these changes Oct 20, 2023

View reviewed changes

alanjhughes approved these changes Oct 21, 2023

View reviewed changes

Kudo merged commit 2f6ba2a into main Oct 23, 2023
18 checks passed

Kudo deleted the @kudo/sqlite-next branch October 23, 2023 06:59

Kudo mentioned this pull request Oct 25, 2023

[sqlite] add android implementation for sqlite/next api #25021

Merged

3 tasks

Kudo added a commit that referenced this pull request Nov 14, 2023

[doc] add sqlite/next doc (#24962)

3624ac9

# Why following up #24812 to update the doc --------- Co-authored-by: Aman Mittal <[email protected]> Co-authored-by: Bartosz Kaszubowski <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[sqlite][ios] introduce sqlite/next #24812

[sqlite][ios] introduce sqlite/next #24812

Kudo commented Oct 10, 2023 •

edited

Loading

linear bot commented Oct 10, 2023

alanjhughes commented Oct 11, 2023

Kudo commented Oct 11, 2023

Kudo commented Oct 12, 2023

tsapeta left a comment

tsapeta Oct 13, 2023

alanjhughes Oct 16, 2023

tsapeta Oct 13, 2023

Kudo Oct 20, 2023

tsapeta Oct 13, 2023

brentvatne Oct 18, 2023 •

edited

Loading

tsapeta Oct 19, 2023

Kudo Oct 19, 2023

Kudo Oct 19, 2023 •

edited

Loading

Kudo commented Oct 19, 2023

Kudo commented Oct 20, 2023

linear bot commented Oct 20, 2023

[sqlite][ios] introduce sqlite/next #24812

[sqlite][ios] introduce sqlite/next #24812

Conversation

Kudo commented Oct 10, 2023 • edited Loading

Why

How

Test Plan

Checklist

linear bot commented Oct 10, 2023

alanjhughes commented Oct 11, 2023

Kudo commented Oct 11, 2023

Kudo commented Oct 12, 2023

tsapeta left a comment

Choose a reason for hiding this comment

tsapeta Oct 13, 2023

Choose a reason for hiding this comment

alanjhughes Oct 16, 2023

Choose a reason for hiding this comment

tsapeta Oct 13, 2023

Choose a reason for hiding this comment

Kudo Oct 20, 2023

Choose a reason for hiding this comment

tsapeta Oct 13, 2023

Choose a reason for hiding this comment

brentvatne Oct 18, 2023 • edited Loading

Choose a reason for hiding this comment

tsapeta Oct 19, 2023

Choose a reason for hiding this comment

Kudo Oct 19, 2023

Choose a reason for hiding this comment

Kudo Oct 19, 2023 • edited Loading

Choose a reason for hiding this comment

Kudo commented Oct 19, 2023

Kudo commented Oct 20, 2023

linear bot commented Oct 20, 2023

Kudo commented Oct 10, 2023 •

edited

Loading

brentvatne Oct 18, 2023 •

edited

Loading

Kudo Oct 19, 2023 •

edited

Loading