Skip to content

Commit

Permalink
docs: update shapes guide. (#1680)
Browse files Browse the repository at this point in the history
  • Loading branch information
thruflo authored Sep 12, 2024
1 parent 65c5122 commit 7a07cde
Show file tree
Hide file tree
Showing 6 changed files with 171 additions and 6 deletions.
3 changes: 3 additions & 0 deletions website/.vitepress/theme/custom.css
Original file line number Diff line number Diff line change
Expand Up @@ -523,6 +523,9 @@ iframe {
.vp-doc blockquote {
margin: 25px 10px 30px;
}
.vp-doc .custom-block {
margin: 24px 0;
}

.custom-block-no-title {
margin-bottom: 10px !important;
Expand Down
3 changes: 3 additions & 0 deletions website/docs/guides/auth.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
---
title: Auth - Guide
description: >-
How to authenticate users and authorize data access with Electric.
outline: deep
---

Expand Down
168 changes: 165 additions & 3 deletions website/docs/guides/shapes.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,175 @@
---
title: Shapes - Guide
description: >-
Shapes are the core primitive for controlling sync in the ElectricSQL system.
outline: deep
---

<script setup>
import SyncShapeJPG from '/static/img/docs/guides/shapes/sync-shape.jpg?url'
import SyncShapePNG from '/static/img/docs/guides/shapes/sync-shape.png?url'
</script>

# Shapes

Shapes are the core primitive for controlling sync in the ElectricSQL system.

Local apps establish shape subscriptions. This syncs data over the [http](/docs/api/http) API from the Electric server onto the local device.
## What is a Shape?

Electric syncs little subsets of your Postgres data into local apps, services and environments. Those subsets are defined using Shapes.

### Little subsets

Imagine a Postgres database in the cloud with lots of data stored in it. It's often impractical or unwanted to sync all of this data over the network onto a local device.

A shape is a way of defining a subset of that data that you'd like to sync into a local app. Defining shapes allows you to sync just the data you want and just the data that's practical to sync onto the local device.

<figure>
<a :href="SyncShapeJPG">
<img :src="SyncShapePNG"
alt="Illustration of syncing a shape"
/>
</a>
</figure>

A client can choose to sync one shape, or lots of shapes. Many clients can sync the same shape. Multiple shapes can overlap.

## Defining shapes

Shapes are defined by:

- a `root_table`, such as `projects`
- a `where` clause, used to filter the rows in that table, such as `status='active'`

> [!IMPORTANT] Limitations
> Shapes are currently single table, whole row only. You can sync all the rows in a table, or a subset of the rows in that table. You can't yet [select columns](#whole-rows) or sync an [include tree](#single-table) without filtering or joining in the client.
### `root_table`

This is the root table of the shape. It must match a table in your Postgres database.

The value can be just a tablename like `projects`, or can be a qualified tablename prefixed by the database schema using a `.` delimiter, such as `foo.projects`. If you don't provide a schema prefix, then the table is assumed to be in the `public.` schema.

### `where` clause

Optional where clause to filter rows in the `root_table`.

This must be a valid [PostgreSQL WHERE clause](https://www.postgresql.org/docs/current/queries-table-expressions.html#QUERIES-WHERE) using SQL syntax, e.g.:

- `title='Electric'`
- `status IN ('backlog', 'todo')`

You can use logical operators like `AND` and `OR` to group multiple conditions, e.g.:

- `title='Electric' OR title='SQL'`
- `title='Electric' AND status='todo'`

Where clauses can only refer to columns in the target row; you can't perform joins or refer to other tables. Where clauses also can't use SQL functions like `count()`.

## Subscribing to shapes

Local clients establish shape subscriptions, typically using [client libraries](/docs/api/clients/typescript). These sync data from the [Electric sync service](/product/electric) into the client using the [HTTP API](/docs/api/http).

The sync service maintains shape subscriptions and streams any new data and data changes to the local
client. In the client, shapes can be held as objects in memory, for example using a [`useShape`](/docs/api/integrations/react) hook, or in a normalised store or database like [PGlite](/product/pglite).

### HTTP

You can sync shapes manually using the
<a href="/openapi.html#/paths/~1v1~1shape~1%7Broot_table%7D/get"
target="_blank">
<code>GET /v1/shape</code></a> endpoint. First make an initial sync request to get the current data for the Shape, such as:

```sh
curl -i 'http://localhost:3000/v1/shape/foo?offset=-1'
```

Then switch into a live mode to use long-polling to receive real-time updates:

```sh
curl -i 'http://localhost:3000/v1/shape/foo?live=true&offset=...&shape_id=...'
```

These requests both return an array of [Shape Log](/docs/api/http#shape-log) entries. You can process these manually, or use a higher-level client.

### Typescript

You can use the [Typescript Client](/docs/api/clients/typescript) to process the Shape Log and materialised it into a `Shape` object for you.

First install using:

```sh
npm i @electric-sql/client
```

Instantiate a `ShapeStream` and materialise into a `Shape`:

```ts
import { ShapeStream, Shape } from '@electric-sql/client'

const stream = new ShapeStream({
url: `http://localhost:3000/v1/shape/foo`,
})
const shape = new Shape(stream)

// Returns promise that resolves with the latest shape data once it's fully loaded
await shape.value
```

You can register a callback to be notified whenever the shape data changes:

```ts
shape.subscribe(shapeData => {
// shapeData is a Map of the latest value of each row in a shape.
})
```

Or you can use framework integrations like the [`useShape`](/docs/api/integrations/react) hook to automatically bind materialised shapes to your components.

See the [Quickstart](/docs/quickstart) and [HTTP API](/docs/api/http) docs for more information.

## Limitations

### Single table

Shapes are currently single table only.

In the [old version of Electric](https://legacy.electric-sql.com/docs/usage/data-access/shapes), Shapes had an include tree that allowed you to sync nested relations. The new Electric has not yet implemented support for include trees.

You can upvote and discuss adding support for include trees here:

- [Shape support for include trees #1608](https://github.com/electric-sql/electric/discussions/1608)

> [!TIP] Include tree workarounds
> There are some practical workarounds you can already use to sync related data, based on subscribing to multiple shapes and joining in the client.
>
> For a one-level deep include tree, such as "sync this project with its issues", you can sync one shape for projects `where="id=..."` and another for issues `where="project_id=..."`.
>
> For multi-level include trees, such as "sync this project with its issues and their comments", you can denormalise the `project_id` onto the lower tables so that you can also sync comments `where="project_id=1234"`.
>
> Where necessary, you can use triggers to update these denormalised columns.
### Whole rows

Shapes currently sync all the columns in a row.

It's not yet possible to select or ignore/mask columns. You can upvote and discuss adding support for selecting columns here:

- [Shape support for selecting columns #1676](https://github.com/electric-sql/electric/discussions/1676)

### Immutable

Shapes are currently immutable.

Once a shape subscription has been started, it's definition cannot be changed. If you want to change the data in a shape, you need to start a new subscription.

You can upvote and discuss adding support for mutable shapes here:

- [Editable shapes #1677](https://github.com/electric-sql/electric/discussions/1677)

<!--
## Performance
The Electric sync service maintains shape subscriptions and streams any new data and data changes onto the local device. In this way, local devices can sync a sub-set of a larger database for interactive local use.
... add links to benchmarks here ...
Local apps ask the server for a specific set of related data that gets synced to the device. The central Postgres instance will often have too much data to fit on any one device, so shapes allow us to sync only a required subset of data onto the device. There is a balance between local data availability and storage usage on the device that is unique to every application, and shapes allow you to balance those properties while maintaining required guarantees.
-->
3 changes: 0 additions & 3 deletions website/electric-api.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,6 @@ paths:
Can be just a tablename, or can be prefixed by the database schema
using a `.` delimiter, such as `foo.issues`. If you don't provide
a schema prefix, then the table is assumed to be in the `public.` schema.
Note that tablenames with special characters may not be parsed correctly
pending https://github.com/electric-sql/electric/issues/185
# Query parameters
- name: offset
in: query
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 7a07cde

Please sign in to comment.