feat: Add lens migration engine to defra #1564

AndrewSisley · 2023-06-09T15:22:24Z

Relevant issue(s)

Known out of scope issues documented with inline todo comments:

I see none of these as blockers for a first release, feel very free to argue with me on that though. Some may be ticked off anyway if I find myself waiting for a while on this PR. Some are quite small and likely quick to fix, but this PR is plenty large already.

Todos (within this PR):

Description

Adds lens as a migration engine to defra.

Migrations are run lazily, on fetch. This includes on updates (locally, and via P2P). The migration result is cached within the datastore. The DAG is never updated to reflect the migration result, including when the migrations are executed during an update. The current schema version of items in the datastore is tracked at the document level.

Commits should be clean, and contain some hopefully handy documentation, although the bulk of the work is in the last, large commit.

Notes

Noted briefly in the todo section, this does make rust, and rust-wasm dependencies of make test, ~~I'll document that before merge~~ as documented within CONTRIBUTING.md. Just in case the question is asked - no I really do not want to write the test modules in Go, Go wasm support is currently awful and it is not worth the hassle at the moment.
The CI build does not pass as a few dependencies are just local paths at the moment. This will change when the Lens repo is open sourced, and the immutable (enumerables) repo is updated. The linter and make test passes on my machine. The change detector will not pass atm, as this PR contains breaking changes that will be documented shortly before merge.
Lens repo is here, if anyone wants a quick link: https://github.com/lens-vm/lens
Performance has taken a bit of a hit - roughly 150µs per fetcher Start call (on my laptop) due to the getHistory func.

As the fetcher is mocked it has zero impact on the test.

islamaliev · 2023-07-21T13:00:18Z

note (bug bash): no way to get schemaVersionID

At the moment there is no easy way to get a schema's version id (that is required to create migrations). I used quick fix made by John on branch https://github.com/sourcenetwork/defradb/tree/jsimnz/feat/schema-list-id-extension that extends client schema list to show schema version ids

islamaliev · 2023-07-21T13:03:24Z

todo (bug bash): improve UX with ~ in path

Path from home doesn’t work. Executed command:

defradb client schema migration set bafkreidovoxkxttybaew2qraoelormm63ilutzms7wlwmcr3xru44hfnta bafkreia4bbxhtqwzw4smby5xsqxv6ptoc6ijc6v3lmnlv66twpfak5gxxq '{"Lenses": [{"Path": "~/dev/wasm/set_default.wasm", "Arguments": {"dst":"verified", "value": true}}]}'

Got

2023-07-21T10:42:38.278+0200, ERROR, cli, logging/logger.go:107, Failed to set schema migration, {"Errors": [{"message":"open ~/dev/wasm/set_default.wasm: no such file or directory"}]}

With a path from root works :

defradb client schema migration set bafkreidovoxkxttybaew2qraoelormm63ilutzms7wlwmcr3xru44hfnta bafkreia4bbxhtqwzw4smby5xsqxv6ptoc6ijc6v3lmnlv66twpfak5gxxq '{"Lenses": [{"Path": "/Users/islam/dev/wasm/set_default.wasm", "Arguments": {"dst":"verified", "value": true}}]}'
2023-07-21T10:42:02.350+0200, INFO, cli, logging/logger.go:98, Successfully set schema migration

If we don't support ~ then I think the message should clearly say that, instaed of "no such file or directory"

islamaliev · 2023-07-21T13:04:48Z

todo (bug bash): remove duplicated output for migrate get

Migration get outputs the result twice:

defradb client schema migration get
2023-07-21T10:59:18.911+0200, INFO, cli, logging/logger.go:98, {"data":{"configuration":[{"SourceSchemaVersionID":"bafkreidovoxkxttybaew2qraoelormm63ilutzms7wlwmcr3xru44hfnta","DestinationSchemaVersionID":"bafkreia4bbxhtqwzw4smby5xsqxv6ptoc6ijc6v3lmnlv66twpfak5gxxq","Lenses":[{"Path":"/Users/islam/dev/wasm/set_default.wasm","Inverse":false,"Arguments":{"dst":"verified","value":true}}]}]}}
2023-07-21T10:59:18.911+0200, INFO, cli, logging/logger.go:98, Successfully got schema migrations, {"Configuration": [{"SourceSchemaVersionID":"bafkreidovoxkxttybaew2qraoelormm63ilutzms7wlwmcr3xru44hfnta","DestinationSchemaVersionID":"bafkreia4bbxhtqwzw4smby5xsqxv6ptoc6ijc6v3lmnlv66twpfak5gxxq","Lenses":[{"Path":"/Users/islam/dev/wasm/set_default.wasm","Inverse":false,"Arguments":{"dst":"verified","value":true}}]}]}

That's not ideal user experience and we should fix it.

islamaliev · 2023-07-21T13:22:57Z

todo (bug bash): fix migration

created a schema:

type Users {
	age: Int
	name: String
}

with schema version bafkreidovoxkxttybaew2qraoelormm63ilutzms7wlwmcr3xru44hfnta

Created a doc:

{
    name: "John"
    age: 27
}

Dockey: bae-88b63198-7d38-5714-a9ff-21ba46374fd1

Patched the schema by adding verified field:

type Users {
	age: Int
	name: String
	verified: Boolean
}

schema version: bafkreia4bbxhtqwzw4smby5xsqxv6ptoc6ijc6v3lmnlv66twpfak5gxxq

Added migration:

defradb client schema migration set bafkreidovoxkxttybaew2qraoelormm63ilutzms7wlwmcr3xru44hfnta bafkreia4bbxhtqwzw4smby5xsqxv6ptoc6ijc6v3lmnlv66twpfak5gxxq '{"Lenses": [{"Path": "/Users/islam/dev/wasm/set_default.wasm", "Arguments": {"dst":"verified", "value": true}}]}'
2023-07-21T10:42:02.350+0200, INFO, cli, logging/logger.go:98, Successfully set schema migration

Queried users with Altair GraphQL client:

query QueryUsers {
  Users(dockey: "bae-88b63198-7d38-5714-a9ff-21ba46374fd1") {
    _key
    name
    age
    verified
    _version {
      cid
      schemaVersionId
    }
  }
}

Got:

{
  "data": [
    {
      "_key": "bae-88b63198-7d38-5714-a9ff-21ba46374fd1",
      "_version": [
        {
          "cid": "bafybeievhpxpptz2y2e4xzutpm2xek4zbvdv25hevt3hv6elxvhysan2t4",
          "schemaVersionId": "bafkreidovoxkxttybaew2qraoelormm63ilutzms7wlwmcr3xru44hfnta"
        }
      ],
      "age": 27,
      "name": "John",
      "verified": null
    }
  ]
}

islamaliev · 2023-07-24T09:00:07Z

note (bug bash): successful (allegedly) migration is not shown with client migration get

Executed migration set command using non-existing source schema version id:

$ defradb client schema migration set bafkreidovoxkxttybaew2qraoelormm63ilutzms7wlwmcr3xru44hfnta bafkreia4bbxhtqwzw4smby5xsqxv6ptoc6ijc6v3lmnlv66twpfak5gxxqw '{"lenses": [{"path": "/Users/islam/dev/wasm/set_default.wasm", "arguments": {"dst":"verified", "value": true}}]}'
2023-07-24T10:52:32.055+0200, INFO, cli, logging/logger.go:98, Successfully set schema migration

Execuged migration get:

$ defradb client schema migration get
2023-07-24T10:52:35.003+0200, INFO, cli, logging/logger.go:98, {"data":{"configuration":[{"SourceSchemaVersionID":"bafkreidovoxkxttybaew2qraoelormm63ilutzms7wlwmcr3xru44hfntae","DestinationSchemaVersionID":"bafkreia4bbxhtqwzw4smby5xsqxv6ptoc6ijc6v3lmnlv66twpfak5gxxq","Lenses":[{"Path":"/Users/islam/dev/wasm/set_default.wasm","Inverse":false,"Arguments":{"dst":"verified","value":true}}]},{"SourceSchemaVersionID":"bafkreidovoxkxttybaew2qraoelormm63ilutzms7wlwmcr3xru44hfnta","DestinationSchemaVersionID":"bafkreia4bbxhtqwzw4smby5xsqxv6ptoc6ijc6v3lmnlv66twpfak5gxxqw","Lenses":[{"Path":"/Users/islam/dev/wasm/set_default.wasm","Inverse":false,"Arguments":{"dst":"verified","value":true}}]}]}}

Formattet json:

{
	"data": {
		"configuration": [{
			"SourceSchemaVersionID": "bafkreidovoxkxttybaew2qraoelormm63ilutzms7wlwmcr3xru44hfntae",
			"DestinationSchemaVersionID": "bafkreia4bbxhtqwzw4smby5xsqxv6ptoc6ijc6v3lmnlv66twpfak5gxxq",
			"Lenses": [{
				"Path": "/Users/islam/dev/wasm/set_default.wasm",
				"Inverse": false,
				"Arguments": {
					"dst": "verified",
					"value": true
				}
			}]
		}, {
			"SourceSchemaVersionID": "bafkreidovoxkxttybaew2qraoelormm63ilutzms7wlwmcr3xru44hfnta",
			"DestinationSchemaVersionID": "bafkreia4bbxhtqwzw4smby5xsqxv6ptoc6ijc6v3lmnlv66twpfak5gxxq",
			"Lenses": [{
				"Path": "/Users/islam/dev/wasm/set_default.wasm",
				"Inverse": false,
				"Arguments": {
					"dst": "verified",
					"value": true
				}
			}]
		}]
	}
}

Observe that just added migration is not there. Looks like it shouldn't be successful.

islamaliev · 2023-07-24T09:19:39Z

todo (bug bash): regard source schema version id or remove it

I entered non-existing schema version id (with "w" in the end) for migration:

$ defradb client schema migration set bafkreidovoxkxttybaew2qraoelormm63ilutzms7wlwmcr3xru44hfnta bafkreia4bbxhtqwzw4smby5xsqxv6ptoc6ijc6v3lmnlv66twpfak5gxxqw '{"lenses": [{"path": "/Users/islam/dev/wasm/set_default.wasm", "arguments": {"dst":"verified", "value": true}}]}'
2023-07-24T11:11:37.568+0200, INFO, cli, logging/logger.go:98, Successfully set schema migration

$ defradb client schema migration get
2023-07-24T11:12:38.420+0200, INFO, cli, logging/logger.go:98, {"data":{"configuration":[{"SourceSchemaVersionID":"bafkreidovoxkxttybaew2qraoelormm63ilutzms7wlwmcr3xru44hfnta","DestinationSchemaVersionID":"bafkreia4bbxhtqwzw4smby5xsqxv6ptoc6ijc6v3lmnlv66twpfak5gxxqw","Lenses":[{"Path":"/Users/islam/dev/wasm/set_default.wasm","Inverse":false,"Arguments":{"dst":"verified","value":true}}]}]}}

Afterwards fetched the document:

{
  "data": [
    {
      "_key": "bae-88b63198-7d38-5714-a9ff-21ba46374fd1",
      "age": 27,
      "name": "John",
      "verified": true
    }
  ]
}

islamaliev · 2023-07-24T10:41:47Z

note (bug bash): client migrate get doesn't show the first migration

I have 2 migrations, but client migrate get shows only the second one:

$ defradb client schema migration get
2023-07-24T12:29:07.769+0200, INFO, cli, logging/logger.go:98, {"data":{"configuration":[{"SourceSchemaVersionID":"bafkreia4bbxhtqwzw4smby5xsqxv6ptoc6ijc6v3lmnlv66twpfak5gxxq","DestinationSchemaVersionID":"bafkreidady7elh62qwextzkyfzzqohfrqvt2hcqhufgoclvfbypsrqx52q","Lenses":[{"Path":"/Users/islam/dev/wasm/remove.wasm","Inverse":false,"Arguments":{"target":"verified"}},{"Path":"/Users/islam/dev/wasm/copy.wasm","Inverse":false,"Arguments":{"dst":"exp","src":"age"}}]}]}}

The first one was:

{
	"data": {
		"configuration": [{
			"SourceSchemaVersionID": "bafkreidovoxkxttybaew2qraoelormm63ilutzms7wlwmcr3xru44hfnta",
			"DestinationSchemaVersionID": "bafkreia4bbxhtqwzw4smby5xsqxv6ptoc6ijc6v3lmnlv66twpfak5gxxqw",
			"Lenses": [{
				"Path": "/Users/islam/dev/wasm/set_default.wasm",
				"Inverse": false,
				"Arguments": {
					"dst": "verified",
					"value": true
				}
			}]
		}]
	}
}

AndrewSisley · 2023-07-24T11:43:26Z

todo (bug bash): regard source schema version id or remove it
I entered non-existing schema version id (with "w" in the end) for migration:

This is by design, and quite important for P2P. Discussed in discord.

note (bug bash): client migrate get doesn't show the first migration

There can only be a single migration between any two versions, the second set will have overwritten the first. This is by design.

islamaliev · 2023-07-24T11:55:15Z

question (bug bash): is it fine to delete _key as part of a migration?

I created a migration:

$ defradb client schema migration set bafkreia4bbxhtqwzw4smby5xsqxv6ptoc6ijc6v3lmnlv66twpfak5gxxq bafkreidady7elh62qwextzkyfzzqohfrqvt2hcqhufgoclvfbypsrqx52q '{"lenses": [{"path": "/Users/islam/dev/wasm/remove.wasm", "arguments": {"target":"_key"}}]}'

Now queries:

query QueryUsers {
  Users {
    _key
    name
    age
    _version {
      schemaVersionId
    }
  }
}

return:

{
  "data": [
    {
      "_key": null,
      "_version": null,
      "age": 27,
      "name": "John",
    },
    {
      "_key": null,
      "_version": null,
      "age": 33,
      "name": "Islam",
    }
  ]
}

AndrewSisley · 2023-07-24T13:44:09Z

question (bug bash): is it fine to delete _key as part of a migration?

It shouldn't break anything I think (worth checking the datastore dump to make sure it is still cached under a non-nil key prefix), but it would be very strange to write/configure a migration that would do such a thing. That said, I'm not sure the nil key value would be cached, and it might be if you re-query (to read from cache) that the key value reappears.

Might be worth adding a set of tests to document whatever behaviour exists here.

islamaliev · 2023-07-25T07:33:13Z

question (bug bash): is invalid migration supposed to be cached?

For this patch:

[{ "op": "add", "path": "/Users/Schema/Fields/-", "value": {"Name": "verified", "Kind": "Boolean"} }]

created a migration:

{"lenses": [{"path": "/Users/islam/dev/wasm/set_default.wasm", "arguments": {"dst":"verified", "value": "invalid"}}]}

On querying document verified field was null.
Then set this migration:

{"lenses": [{"path": "/Users/islam/dev/wasm/set_default.wasm", "arguments": {"dst":"verified", "value": 5}}]}

Field verified was still null
Then set this migration:

{"lenses": [{"path": "/Users/islam/dev/wasm/set_default.wasm", "arguments": {"dst":"verified", "value": false}}]}

Field verified was still null!
From the user's perspective that doesn't look healthy.

$ defradb client schema migration get
2023-07-25T09:18:37.778+0200, INFO, cli, logging/logger.go:98, {"data":{"configuration":[{"SourceSchemaVersionID":"bafkreidovoxkxttybaew2qraoelormm63ilutzms7wlwmcr3xru44hfnta","DestinationSchemaVersionID":"bafkreia4bbxhtqwzw4smby5xsqxv6ptoc6ijc6v3lmnlv66twpfak5gxxq","Lenses":[{"Path":"/Users/islam/dev/wasm/set_default.wasm","Inverse":false,"Arguments":{"dst":"verified","value":false}}]}]}}

islamaliev · 2023-07-25T10:39:58Z

note (bug bash): linking a relational object using alias

Schema:

# Schema ID: bafkreibaextmibnllnex7ewwn5ednh6r3znil55mljbl2t3gjgh6unxwra
# Version ID: bafkreibaextmibnllnex7ewwn5ednh6r3znil55mljbl2t3gjgh6unxwra
type Companies {
	employees: [Users]
	name: String
}

# Schema ID: bafkreia7zqb4jlmjqdksyqxhfimwgxotp3qgflfku7nhwkayy7mr4ojflq
# Version ID: bafkreia7zqb4jlmjqdksyqxhfimwgxotp3qgflfku7nhwkayy7mr4ojflq
type Users {
	company: Companies
	name: String
}

Added users.
Added (via patching) verified field to Users.
Set migration for Users:

$ defradb client schema migration set bafkreia7zqb4jlmjqdksyqxhfimwgxotp3qgflfku7nhwkayy7mr4ojflq bafkreiazbfbzel5iixrhgleqrnbb7u6sqkqav7oh54x4qfiguo6eofllzq '{"lenses": [{"path": "/Users/islam/dev/wasm/set_default.wasm", "arguments": {"dst":"company", "value": "bae-de8c99bf-ee0e-5655-8a72-919c2d459a30"}}]}'

Upon requesting a user whose company value was null observed the same:

{
  "data": [
    {
      "_key": "bae-3f1174ba-d9bc-5a6a-b0bc-8f19581f199d",
      "company": null,
      "name": "Islam"
    },
    {
      "_key": "bae-f456e9a1-d539-5ce4-94d7-a2f008df34b4",
      "company": {
        "_key": "bae-de8c99bf-ee0e-5655-8a72-919c2d459a30",
        "name": "Source"
      },
      "name": "John"
    }
  ]
}

Note: when using company_id in lens config it works

AndrewSisley · 2023-07-25T12:36:44Z

linking a relational object using alias

I've added a ticket for this: #1710

islamaliev · 2023-07-26T06:51:43Z

todo (bug bash): fix copying into _key

The setup up the same as above with one Users collection.
Created a migration:

{"lenses": [{"path": "/Users/islam/dev/wasm/copy.wasm", "arguments": {"src":"age", "dst": "_key"}}]}

Queried users and got:

{
  "errors": [
    "invalid key string"
  ],
  "data": null
}

## Relevant issue(s) Resolves sourcenetwork#1448 sourcenetwork#1556 Adds lens as a migration engine to defra. Migrations are run lazily, on fetch. This includes on updates (locally, and via P2P). The migration result is cached within the datastore. The DAG is never updated to reflect the migration result, including when the migrations are executed during an update. The current schema version of items in the datastore is tracked at the document level. Commits should be clean, and contain some hopefully handy documentation, although the bulk of the work is in the last, large commit.

AndrewSisley added feature New feature or request area/schema Related to the schema system action/no-benchmark Skips the action that runs the benchmark. labels Jun 9, 2023

AndrewSisley added this to the DefraDB v0.6 milestone Jun 9, 2023

AndrewSisley self-assigned this Jun 9, 2023

AndrewSisley mentioned this pull request Jun 9, 2023

feat: Expand enumerable types (queue, socket) sourcenetwork/immutable#7

Merged

AndrewSisley force-pushed the 1448-lens-in-defra branch 24 times, most recently from 96d12e1 to fee3881 Compare June 15, 2023 17:13

AndrewSisley added 12 commits July 13, 2023 06:02

PR FIXUP - Fix non-unique update mock test

c7d8f5b

PR FIXUP - Remove misleading save call

1aac0d9

As the fetcher is mocked it has zero impact on the test.

PR FIXUP - Fix typo

f0f55ff

PR FIXUP - Fix typo

cd55fb7

PR FIXUP - Add MIT License to test lens modules

548dbb6

PR FIXUP - Always update ds doc version if migrated

5156255

PR FIXUP - Add missing mocks to make file

18057e3

PR FIXUP - Fetch all fields if migrations registered

4341268

PR FIXUP - Remove lens prefix for registery props

0557fe6

PR FIXUP - Rename warehouse stuff

bf95056

REBASE FIXUP

6fb6415

PR FIXUP - Wrap close errors

09f6825

AndrewSisley force-pushed the 1448-lens-in-defra branch from 947ad88 to 09f6825 Compare July 13, 2023 10:05

AndrewSisley merged commit 5d74fe2 into sourcenetwork:develop Jul 13, 2023
10 checks passed

AndrewSisley deleted the 1448-lens-in-defra branch July 13, 2023 10:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add lens migration engine to defra #1564

feat: Add lens migration engine to defra #1564

AndrewSisley commented Jun 9, 2023 •

edited

Loading

islamaliev commented Jul 21, 2023 •

edited

Loading

islamaliev commented Jul 21, 2023 •

edited

Loading

islamaliev commented Jul 21, 2023 •

edited

Loading

islamaliev commented Jul 21, 2023

islamaliev commented Jul 24, 2023

islamaliev commented Jul 24, 2023

islamaliev commented Jul 24, 2023

AndrewSisley commented Jul 24, 2023

islamaliev commented Jul 24, 2023 •

edited

Loading

AndrewSisley commented Jul 24, 2023

islamaliev commented Jul 25, 2023

islamaliev commented Jul 25, 2023 •

edited

Loading

AndrewSisley commented Jul 25, 2023

islamaliev commented Jul 26, 2023

feat: Add lens migration engine to defra #1564

feat: Add lens migration engine to defra #1564

Conversation

AndrewSisley commented Jun 9, 2023 • edited Loading

Relevant issue(s)

Known out of scope issues documented with inline todo comments:

Todos (within this PR):

Description

Notes

islamaliev commented Jul 21, 2023 • edited Loading

islamaliev commented Jul 21, 2023 • edited Loading

islamaliev commented Jul 21, 2023 • edited Loading

islamaliev commented Jul 21, 2023

islamaliev commented Jul 24, 2023

islamaliev commented Jul 24, 2023

islamaliev commented Jul 24, 2023

AndrewSisley commented Jul 24, 2023

islamaliev commented Jul 24, 2023 • edited Loading

AndrewSisley commented Jul 24, 2023

islamaliev commented Jul 25, 2023

islamaliev commented Jul 25, 2023 • edited Loading

AndrewSisley commented Jul 25, 2023

islamaliev commented Jul 26, 2023

AndrewSisley commented Jun 9, 2023 •

edited

Loading

islamaliev commented Jul 21, 2023 •

edited

Loading

islamaliev commented Jul 21, 2023 •

edited

Loading

islamaliev commented Jul 21, 2023 •

edited

Loading

islamaliev commented Jul 24, 2023 •

edited

Loading

islamaliev commented Jul 25, 2023 •

edited

Loading