11.0.0 beta - Earthstar × Willow

Good morning! The day for an Willow-powered Earthstar 11 beta is finally here. This is a big experiment: a new protocol underpinning everything, complementary new APIs, and of course, the greatest adventure of them all: getting it all to work in JavaScript.

Another experiment is how Earthstar will be distributed. A beta release of @earthstar/earthstar is available from JSR. https://jsr.io/@earthstar/[email protected]. You can find installation instructions for your favourite runtime (yes, Node too) on there. Hopefully JSR will save us from lots of cross-runtime publishing headaches, let’s see.

The README is very much in progress (https://github.com/earthstar-project/earthstar/tree/willow), but there is already very descriptive API documentation available in your IDE and viewable here: https://jsr.io/@earthstar/[email protected]/doc. A great place to start is the Peer class: https://jsr.io/@earthstar/[email protected]/doc/~/Peer

This beta phase serves the purpose of making sure @earthstar/earthstar can really be used to build apps for the browser, Deno, and Node; that the APIs are as ergonomic as they should be; and to get the docs up to snuff. Your feedback — no matter how nitpicky — is now key! Issues, ideas, and PRs are all welcome on the Earthstar repo.

v10.2.2 - Dependency fixes

This is a patch release which fixes two dependencies that began to fail: better-sqlite3 and fast-stable-json-stringify. The former was breaking NPM installs, the latter was breaking Deno installs. Now both are fixed.

v10.2.1 Fixed invitation redemption in Chrome and Firefox

This release fixes a bug where invitation URLS could not be redeemed in Chrome or Firefox (thanks @tripledoublev!)

v10.2.0: LAN peer discovery and sync

This is a feature release adding APIs to automatically discover and sync with peers on the local network. Automatically find and sync with friends at a café, or with your own devices at home.

// Start a new discovery service, and advertise ourselves as "My Computer".
const discoveryLan = new DiscoveryLAN({ name: "My Computer" });

// Wait for new discovery service events...
for await (const discoveryEvent of peer.discover(discoveryLan)) {
  if (discoveryEvent.kind === "PEER DISCOVERED") {
    // We'll sync with everyone we find!
    discoveryEvent.sync();
  }
}

Discovery is powered by dns_sd, and sync itself is powered by a new fully encrypted TCP transport driver.

• (Feature) Added Peer.discover, a new method for automatically discovering other peers. How it does that it left to the DiscoveryService you provide it.
• (Feature) Added DiscoveryLAN, a discovery service for finding and advertising yourself to other peers on the local network. Works for both Node and Deno, requires --unstable on Deno.

v10.1.0: Invitations

This is a feature release which adds support for invitation URLs.

Copying and pasting many addresses, secrets, and URLs is tedious. Copying and pasting a single address which your app already knows what to do with is much better.

They look like this:

earthstar://+shareaddress.xxx/?invite&server=https://server.com&server=https://server2.com&secret=bxxx&v=2

• (Feature) Added SharedSettings#redeemInvitationURL, which adds the share, secret, and servers from an Earthstar invitation URL.
• (Feature) Added createInvitationURL and parseInvitationURL utilities for creating and parsing Earthstar invitation URLs.
• (Fix) parseShareAddress now validates share addresses more strictly so as to ensure their suffix is a pubkey.
• (Fix) Deprecate generateShareAddress
• (Fix) Fixed some issues ExtensionServerSettings when ephemeral and
  non-ephemeral settings for a hosted share existed at the same time.
• (Chore) Update range_reconcile to 1.0.2, which makes sync slightly more
  efficient.

v10.0.2 - Updated Deno SqliteFFI driver, animal story

Fixes

• DocDriverSqliteFfi has been updated to use https://deno.land/x/[email protected] which has compatibility with Deno 1.30.0 and up. This driver uses unstable APIs and will not work with previous versions of Deno.

Docs

• Updated the example 'animal story' script to showcase Earthstar's ephemeral document features. Thank you @johanbove!
• Updated the README to make it clear that certain APIs like ReplicaDriverWeb are only available through different namespaces in the NPM distribution.

Chores

• Upgraded range-reconcile to version 1.0.1.

v10.0.1: Syncing fixes and improved errors

This is a patch release focused on resolving errors encountered during syncing, and making other issues encountered during syncing easier to diagnose.

Thank you to everyone who helped me put together the clues to find these bugs!

• (Fix) - Peers will now only initiate a single transfer for many documents with the same attachment, fixing a case which could cause syncing to hang indefinitely.
• (Fix) - Peers will now attempt to download attachments for documents which they already possess prior to syncing but are missing attachments for.
• (Improvement) - Syncer will cancel itself if it does not receive anything from the other peer within a ten second window.
• (Improvement) - Syncer will cancel an attachment transfer if it doesn't receive anything from the other side within a ten second window.
• (Improvement) - Warn in the console when a replica could not ingest an attachment during sync.
• (Improvement) - Better error messages for web syncing failures (e.g. 404, wrong endpoint).

v10: Attachments, Read-only shares, and Efficient Sync

v10.0.0 - Squirrel

🐿 Cute, fast, and likes to store things.

This is a major release which introduces attachments, share keypairs, efficient
sync, and much much more. It is the biggest release of Earthstar ever.

As such, this version breaks compatibility with previous versions of Earthstar.

Here are the headline features:

• Attachments. When a new document is written to a Replica you can attach
  arbitary binary data to it. This can be used for sharing large images, music,
  video, anything. There is no size limit.
• Shares with granular read / write access. Share addresses are now the
  public key of a share keypair. The public key grants discovery and read
  access, the secret key grants write access to the replica.
• Efficient sync. Syncing has been completely overhauled to use a new
  efficient reconclition mechanism powered by
  range-reconcile and
  push-pull-push-multicast trees.

The server APIs have also been moved into this repo.

In addition to new features, many APIs have been tweaked or changed entirely.
Please see the API documentation and the README to see what these new API
changes are like.

Server

• Moved existing server APIs into the core module, available as Server.

Peer

• Added Peer.onReplicasChange
• Added Peer.onSyncersChange

Replica

• Significantly improved the performance of querying documents.
• Replicas now can create documents with attachments with Replica.set
• Added Replica.ingestAttachment
• Added Replica.getAttachment
• Added Replica.addAttachments
• Added Replica.wipeDocAtPath
• Added Replica.getEventStream
• Added Replica.getQueryStream
• Added Replica.onEvent
• Added MultiformatReplica, a Replica which is able to read, write, and sync
  documents of different formats.
• Added FormatEs5, which supports share keypairs and attachments
• Added ReplicaDriverWeb
• Added ReplicaDriverFs
• Added RelpicaDriverMemory
• Added DocDriverSqliteFFI, which uses an FFI implemetation of Sqlite.
  Requires the --unstable flag on Deno.
• Updated syncReplicaAndFsDir to use attachments for large files.
• ReplicaCache now has attachment methods

Syncing

• Added PartnerLocal, for syncing with local peers.

• Added PartnerWebServer, for syncing with servers.

• Added PartnerWebClient, for syncing with web clients.

• Removed earthstar_streaming_rpc as a dependency.

Queries

• Removed the contentLength options on QueryFilter.

• Removed QueryFollower. Use Replica.getQueryStream instead.

• queryByTemplateAsync and queryByGlobAsync have had the redundant async
  taken out of their name.

Cryptography

• Added Crypto.generateShareKeypair
• Added CryptoDriverSodium which uses a WASM version of libsodium for very
  fast operations. This is now the default driver on Deno.
• Updated CryptoDriverNoble to use a new, faster, audited version.

Other

• Added a new SharedSettings class for easily saving and retrieving an author
  keypair, shares and secrets, and favourite servers.
• Added parseAuthorOrShareAddress
• Added a new minified web bundle, available from
  https://cdn.earthstar-project.org/js/earthstar.web.v10.0.0.js
• Added ARCHITECTURE.md
• Added CONTRIBUTING.md
• Added CODE_OF_CONDUCT.md

v8.2.4

I've made a number of changes which specifically target Earthstar's performance in the context of dynamic UIs e.g. React apps.

Changes since 8.10:

New NPM API

• Breaknig: moved modules with environmental assupmtions (e.g. ReplicaDriverSqlite, ReplicaDriverLocalStorage) into earthstar/browser and earthstar/node import subpaths.

Replica

• Share addresses are now validated when a Replica is being instantiated.

Syncing

• SyncCoordinator now only has one active pull from another peer at a time
• The maxLocalIndex of other peers is now better preserved across async calls, preventing refetches

ReplicaDriverIndexedDB

• ReplicaDriverIndexedDB now waits until the underlying DB is ready when querying docs
• Fixed an issue where the driver's maxLocalIndex was not being correctly instantiated

ReplicaDriverLocalStorage

• Fixed an issue where the driver's maxLocalIndex was not being correctly instantiated

ReplicaCache

• Added a close method to ReplicaCache which closes all the QueryFollowers during its usage
• Added a onCacheUpdatedWrapper arg to ReplicaCache's constructor, allowing all callbacks to be wrapped within a function call.
• ReplicaCache now only fires cache updates when the results of a query do not match with the cache
• ReplicaCache now reuses QueryFollowers more efficiently

v8.1.0

Previously it was possible to create a Replica with an invalid share address.

const replica = new Replica("PEANUTS.123", FormatValidatorEs4, driver);
// No complaints until you try to set or sync docs.

Now it will throw an error.