Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Virtualize WAL methods #53

Merged
merged 16 commits into from
Dec 21, 2022
Merged

Virtualize WAL methods #53

merged 16 commits into from
Dec 21, 2022

Commits on Dec 13, 2022

  1. pager, wal: virtualize WAL methods

    This preparatory commit moves all WAL routines to a virtual table.
    Also, a helper libsql_open() function is provided. It allows passing
    a new parameter - name of the custom WAL methods implementation.
    psarna committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    519a33b View commit details
    Browse the repository at this point in the history
  2. wal: add bUsesShm flag to declare relying on shared memory

    Before enabling WAL journaling, libSQL detects whether WAL
    is fully supported. Historically it meant either operating
    in exclusive mode or supporting shared memory locks in the
    underlying VFS, but it may not be the case with custom WAL
    implementations. Thus, custom WAL methods can declare whether
    they rely on shared memory - if not, it will also not get
    verified when checking for WAL support.
    psarna committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    c927cac View commit details
    Browse the repository at this point in the history
  3. wal: Virtualize pathname handling

    It makes little sense to store the WAL file name
    if the virtual WAL implementation is not based
    on a single file. Therefore, WAL pathname handling
    is also virtualized, with the original implementation
    still producing a <dbname>-wal file.
    psarna committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    423512f View commit details
    Browse the repository at this point in the history
  4. ext: add a stub for custom WAL methods

    This commit adds a stub implementation of custom WAL methods
    in ext/vwal subdirectory. It can be used as a starting point
    for implementing custom WAL routines.
    psarna committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    284c370 View commit details
    Browse the repository at this point in the history
  5. test: add a Rust test for virtual WAL

    The test case registers a new naive virtual WAL coded on top
    of Rust's std::collections::HashMap. The WAL implementation
    only covers reading and writing pages, without checkpointing
    or savepoints, but it's enough to validate that the virtual
    method table works. After registering custom WAL, a few simple
    operations are performed on the database to validate that
    pages were indeed stored and retrieved properly.
    For extra logs, run the test with -- --nocapture.
    psarna committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    ec756d9 View commit details
    Browse the repository at this point in the history
  6. shell: add .walmethodslist

    The command is heavily inspired by .vfslist
    and lists all the registered custom WAL methods.
    psarna committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    7e99eda View commit details
    Browse the repository at this point in the history
  7. wal: add autoinitialization to libsql_wal_methods* methods

    These are customarily run early, before a call to libsql_open,
    so it makes sense to auto-initialize.
    psarna committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    8ca58ac View commit details
    Browse the repository at this point in the history
  8. ext: allow registering WAL methods via extension API

    With this patch applied, WAL methods can be registered
    from a loadable extension module dynamically.
    psarna committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    5170e4e View commit details
    Browse the repository at this point in the history
  9. Expose wal_methods_unregister in the extension API as well

    Not critical, as WAL methods would customarily live as long
    as the program that runs it, but it's good practice to be able
    to unregister during a cleanup.
    psarna committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    47e59b7 View commit details
    Browse the repository at this point in the history
  10. Add a pre-open-main-db-file hook

    WAL is open lazily on first access, while it is useful to be able
    to set up ground for it even before the main db file is open.
    This optional hook allows it.
    psarna committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    798539e View commit details
    Browse the repository at this point in the history
  11. vwal: introduce iVersion for being future-proof

    Similarly to how other interfaces work, the version number in WAL methods
    lets the user know which functions are supported, and which aren't yet.
    psarna committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    7c59954 View commit details
    Browse the repository at this point in the history
  12. test: update the virtual WAL Rust test

    The test now properly sets the iVersion and pre-main-db-open hook.
    psarna committed Dec 13, 2022
    Configuration menu
    Copy the full SHA
    8419336 View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2022

  1. wal remove ifdefs from libsql_wal_methods

    ABI should be consistent regardless of the compilation options,
    so the optional functions are in there anyway - they can be simply
    set to nulls if the user did not compile support for them in libSQL.
    psarna committed Dec 14, 2022
    Configuration menu
    Copy the full SHA
    929f9ba View commit details
    Browse the repository at this point in the history
  2. wal: add optional pointer to private methods data in struct Wal

    It will be useful for any state that custom WAL methods could
    potentially have.
    psarna committed Dec 14, 2022
    Configuration menu
    Copy the full SHA
    486dd38 View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2022

  1. attach: add missing header

    In amalgamation mode it compiled just fine, but otherwise
    it relies on the wal.h header now.
    psarna committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    c150084 View commit details
    Browse the repository at this point in the history
  2. doc: add a paragraph on virtual WAL

    The paragraph briefly explains the newly introduced mechanism.
    psarna committed Dec 20, 2022
    Configuration menu
    Copy the full SHA
    045225e View commit details
    Browse the repository at this point in the history