-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Deprecate peapod, provide migration to fstree #3013
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3013 +/- ##
==========================================
+ Coverage 22.82% 22.84% +0.01%
==========================================
Files 790 791 +1
Lines 58362 58603 +241
==========================================
+ Hits 13323 13389 +66
- Misses 44157 44316 +159
- Partials 882 898 +16 ☔ View full report in Codecov by Sentry. |
694842f
to
b189b3f
Compare
b189b3f
to
649777b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am almost sure we also want to migrate some metabase indexes in this util, otherwise i do not know who should do it.
See
neofs-node/pkg/local_object_storage/blobstor/get.go
Lines 19 to 32 in ba1d639
if prm.StorageID == nil { | |
for i := range b.storage { | |
res, err := b.storage[i].Storage.Get(prm) | |
if err == nil || !errors.As(err, new(apistatus.ObjectNotFound)) { | |
return res, err | |
} | |
} | |
return common.GetRes{}, logicerr.Wrap(apistatus.ObjectNotFound{}) | |
} | |
if len(prm.StorageID) == 0 { | |
return b.storage[len(b.storage)-1].Storage.Get(prm) | |
} | |
return b.storage[0].Storage.Get(prm) |
neofs-node/pkg/local_object_storage/blobstor/peapod/peapod.go
Lines 422 to 425 in ba1d639
return common.PutRes{ | |
StorageID: storageID, | |
}, err | |
} |
return common.GetRes{Object: obj, RawData: data}, nil |
Linter feels bad. |
649777b
to
34fc018
Compare
After migration from Peapod to Fstree, the config may have 1 substorage, if not already migrated, there are 2 substorages. Refs #2924. Signed-off-by: Andrey Butusov <[email protected]>
Also suppress linter in `neofs-lens`. Refs #2924. Signed-off-by: Andrey Butusov <[email protected]>
34fc018
to
7a17640
Compare
type epochState struct{} | ||
|
||
func (s epochState) CurrentEpoch() uint64 { | ||
return 0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure what can be affected by this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How can I easily get the epoch state in another way? The metabase requires it to be set.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only via network interaction. Maybe it's not a problem in our case (updating storage ID doesn't require epoch IIRC). Or maybe we can get some latest known one from meta or local state.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can get LastResyncEpoch from metabase:
neofs-node/pkg/local_object_storage/metabase/last_resync_epoch.go
Lines 12 to 14 in 45802c6
// ReadLastResyncEpoch reads from db last epoch when metabase was resynchronized. | |
// If id is missing, returns 0, nil. | |
func (db *DB) ReadLastResyncEpoch() (epoch uint64, err error) { |
But it will be necessary to recreate it after receiving an epoch.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's somewhat different in meaning and it's a new one, 0.43.0 DB doesn't have it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why do you need a precise epoch? just pick an epoch that does not claim anything about object expiration and allows GET and PUT freely, that is 0
. @roman-khimov, what bothers you?
How can I easily get the epoch state in another way? The metabase requires it to be set.
i did this and i did not know there would be so many utils that just open metabases and look at them. normal metabase always has to know the current epoch (via internal state or via explicit parameter to every operation)
7a17640
to
bfe2f8e
Compare
cmd/peapod-to-fstree/main.go
Outdated
} | ||
|
||
srcPath := *nodeCfgPath | ||
ss := strings.Split(srcPath, ".") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i would recommend smth like
extension := filepath.Ext(srcPath)
name := srcPath[0:len(srcPath)-len(extension)]
dots are not prohibited in file names
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In general, you can just do + ".migrated"
and that's it. srcPath
can be anything, with many dots, without dots, whatever.
bfe2f8e
to
39f1a79
Compare
Add `cmd/peapod-to-fstree` application which accepts YAML configuration file of the storage node, for each configured shard, overtakes data from Peapod to FSTree that has already been configured and updates metabase `StorageId` indexes. The tool is going to be used for phased and safe rejection of the Peapod and the transition to FSTree. Closes #2924. Signed-off-by: Andrey Butusov <[email protected]>
39f1a79
to
4fb969a
Compare
Closes #2924.
Is there anything else I should do in this pr?