kvdb: refactor as preparation for DB migration command in lndinit #5561

guggero · 2021-07-23T15:43:35Z

EDIT: The main migration tool has moved to lightninglabs/lndinit#21 (still not ready for production/mainnet use!)

The code in this PR is now purely a refactor to allow lndinit to access the required library code.

bhandras

Awesome!!! ⚡ Just a quick glance, will do a more thorough review later!

bhandras · 2021-09-24T08:16:46Z

cmd/lndinit/cmd_migrate_db.go

+			return nil
+		}
+
+		err := dest.Put(k, v)


I think we're missing the sequence migration in this case.

bhandras · 2021-09-24T08:18:52Z

cmd/lndinit/cmd_migrate_db.go

+	ctx, cancel := context.WithTimeout(ctx, etcdTimeout)
+	defer cancel()
+
+	_, err := cli.Put(ctx, key, value)


One quick low hanging speedup is to gather key/values to a buffer then when the buffer hits say 100-1000 items then flush it to etcd in one txn. It's doable since the individual keys don't depend on each other.

bhandras · 2021-09-24T08:21:03Z

cmd/lndinit/cmd_migrate_db.go

+		)
+	})
+	if err != nil {
+		return err


Not sure but perhaps we could delete all keys at this point? Just to avoid leaving any junk in the DB.

guggero · 2021-12-07T13:02:07Z

!lightninglabs-deploy mute 2022-Feb-01

lightninglabs-deploy · 2022-02-01T00:37:37Z

@guggero, remember to re-request review from reviewers when ready

guggero · 2022-02-01T08:40:35Z

!lightninglabs-deploy mute

djkazic · 2022-03-03T15:02:26Z

Is this safe to use for an existing bolt lnd installation?

guggero · 2022-03-05T11:09:06Z

Is this safe to use for an existing bolt lnd installation?

No, this hasn't been tested sufficiently and isn't recommended to be used (which is also the main reason for this not to be merged yet).

kiwiidb · 2022-07-04T12:02:04Z

Hey there,

I've compiled this code and tried to test it out on one of our regtest nodes. However I've failed to do a migration due to the fact that the current channeldb version is already 27, while this version of lndinit is still compiled with channeldb version 23.

2022-07-04 11:47:10.431 LNDINIT: Checking tombstone marker on source DB
2022-07-04 11:47:10.431 LNDINIT: Checking DB version of source DB
2022-07-04 11:47:10.432 LNDINIT: Runtime error: refusing to migrate source database with version 27 while latest known DB version is 23; please upgrade the DB before using the data migration tool

I'd love to test out the migration (on regtest) if these dependencies could be brought up-to-date.

guggero · 2022-07-04T14:43:39Z

Done, @kiwiidb see lightninglabs/lndinit#21.

Please don't use this for production/mainnet data!

guggero · 2022-07-04T14:53:10Z

This is now just basic refactoring code, I think we can review and merge, @bhandras.

bhandras

LGTM ✨

kvdb/etcd/bucket.go

channeldb/meta.go

yyforyongyu · 2022-07-07T09:54:50Z

channeldb/meta.go

+	}
+
+	key := markerBucket.Get(markerKey)
+	if len(key) == 0 {


I think we need to check key against nil here if we want to validate the markerKey's existence. Checking len(key) == 0 seems to have a different meaning as it's checking what's stored, not whether it's stored. Also nit but we may rename key to value or data here as it's the value we get from the bucket...

You are correct, there is a difference between an empty slice and nil. But in our case it doesn't matter which it is, it just means the marker wasn't added correctly. So both cases are equivalent in this case. Added a comment to clarify and also renamed the variable.

channeldb/meta.go

yyforyongyu

Nice set of unit tests! linter complaints, otherwise LGTM💡

yyforyongyu · 2022-10-12T18:00:28Z

channeldb/meta_test.go

+
+	// Only adding the marker bucket should not be enough to be counted as
+	// a marker, we explicitly also want the value to be set.
+	err = db.Update(func(tx walletdb.ReadWriteTx) error {


In order for us to be able to re-use the code for initializing an etcd client connection, we extract and export the NewEtcdClient function.

We'll need to know whether a database was migrated to the latest version in our upcoming data migration tool. To be able to determine the current version of a DB and the total number of migrations in existence, we need to export some of the functions in channeldb. We also add some helper functions for adding tombstone and other markers to a database.

To avoid running a channel DB that was successfully migrated to another system by accident, we check if there is a tombstone marker before we open the DB for use.

For the data migration tool we need to know what database namespaces there are so we can loop over them.

guggero mentioned this pull request Jul 23, 2021

kvdb: migrate between different backends #5524

Closed

guggero added this to the v0.14.0 milestone Jul 23, 2021

guggero added database Related to the database/storage of LND kvdb migration labels Jul 23, 2021

guggero mentioned this pull request Sep 6, 2021

In-memory graph cache for faster pathfinding #5642

Merged

guggero force-pushed the database-migration-lndinit branch 2 times, most recently from 5773d84 to 7702c90 Compare September 23, 2021 14:24

guggero marked this pull request as ready for review September 23, 2021 14:26

guggero requested review from bhandras and Roasbeef September 23, 2021 14:26

guggero force-pushed the database-migration-lndinit branch from 7702c90 to e0850d0 Compare September 23, 2021 14:34

bhandras reviewed Sep 24, 2021

View reviewed changes

Roasbeef modified the milestones: v0.14.0, v0.15.0 Sep 27, 2021

Roasbeef removed their request for review September 29, 2021 03:46

joostjager mentioned this pull request Sep 29, 2021

kvdb: add postgres #5366

Merged

Roasbeef modified the milestones: v0.15.0, v0.16.0 Apr 12, 2022

thinkmassive mentioned this pull request May 6, 2022

How to migrate from bolt db to postgres? #6507

Closed

guggero force-pushed the database-migration-lndinit branch from e0850d0 to 645239a Compare July 4, 2022 14:31

guggero changed the title ~~kvdb: migrate data between different backends~~ kvdb: refactor as preparation for DB migration command in lndinit Jul 4, 2022

guggero force-pushed the database-migration-lndinit branch from 645239a to bd95d5c Compare July 4, 2022 14:51

guggero requested a review from bhandras July 4, 2022 14:53

bhandras approved these changes Jul 4, 2022

View reviewed changes

guggero requested a review from yyforyongyu July 5, 2022 09:15

yyforyongyu reviewed Jul 7, 2022

View reviewed changes

guggero force-pushed the database-migration-lndinit branch from bd95d5c to dd03017 Compare September 29, 2022 10:32

guggero requested a review from yyforyongyu September 29, 2022 10:33

yyforyongyu reviewed Sep 29, 2022

View reviewed changes

channeldb/meta.go Show resolved Hide resolved

channeldb/meta.go Outdated Show resolved Hide resolved

channeldb/meta.go Show resolved Hide resolved

guggero force-pushed the database-migration-lndinit branch from dd03017 to b420e6d Compare September 30, 2022 12:16

guggero requested a review from yyforyongyu September 30, 2022 12:16

guggero force-pushed the database-migration-lndinit branch from b420e6d to c1051d2 Compare September 30, 2022 12:17

yyforyongyu reviewed Oct 12, 2022

View reviewed changes

guggero added 8 commits October 13, 2022 09:37

etcd: fix typos

98f359c

etcd: extract NewEtcdClient into exported function

d576184

In order for us to be able to re-use the code for initializing an etcd client connection, we extract and export the NewEtcdClient function.

etcd: add SequenceKey function

ee9d6f2

channeldb: remove unused tx argument

28c8e8b

channeldb: check for tombstone before opening DB

fc2d652

To avoid running a channel DB that was successfully migrated to another system by accident, we check if there is a tombstone marker before we open the DB for use.

lncfg: export database namespace names

696758a

For the data migration tool we need to know what database namespaces there are so we can loop over them.

docs: add release notes

4e42ef0

guggero force-pushed the database-migration-lndinit branch from c1051d2 to 4e42ef0 Compare October 13, 2022 07:45

guggero merged commit 4b82717 into lightningnetwork:master Oct 13, 2022

guggero deleted the database-migration-lndinit branch October 13, 2022 08:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kvdb: refactor as preparation for DB migration command in lndinit #5561

kvdb: refactor as preparation for DB migration command in lndinit #5561

guggero commented Jul 23, 2021 •

edited

Loading

bhandras left a comment

bhandras Sep 24, 2021

bhandras Sep 24, 2021

bhandras Sep 24, 2021

guggero commented Dec 7, 2021

lightninglabs-deploy commented Feb 1, 2022

guggero commented Feb 1, 2022

djkazic commented Mar 3, 2022

guggero commented Mar 5, 2022

kiwiidb commented Jul 4, 2022

guggero commented Jul 4, 2022

guggero commented Jul 4, 2022

bhandras left a comment

yyforyongyu Jul 7, 2022

guggero Sep 29, 2022

yyforyongyu left a comment

yyforyongyu Oct 12, 2022

kvdb: refactor as preparation for DB migration command in lndinit #5561

kvdb: refactor as preparation for DB migration command in lndinit #5561

Conversation

guggero commented Jul 23, 2021 • edited Loading

bhandras left a comment

Choose a reason for hiding this comment

bhandras Sep 24, 2021

Choose a reason for hiding this comment

bhandras Sep 24, 2021

Choose a reason for hiding this comment

bhandras Sep 24, 2021

Choose a reason for hiding this comment

guggero commented Dec 7, 2021

lightninglabs-deploy commented Feb 1, 2022

guggero commented Feb 1, 2022

djkazic commented Mar 3, 2022

guggero commented Mar 5, 2022

kiwiidb commented Jul 4, 2022

guggero commented Jul 4, 2022

guggero commented Jul 4, 2022

bhandras left a comment

Choose a reason for hiding this comment

yyforyongyu Jul 7, 2022

Choose a reason for hiding this comment

guggero Sep 29, 2022

Choose a reason for hiding this comment

yyforyongyu left a comment

Choose a reason for hiding this comment

yyforyongyu Oct 12, 2022

Choose a reason for hiding this comment

guggero commented Jul 23, 2021 •

edited

Loading