Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

meta: support dir stream #5162

Merged
merged 13 commits into from
Oct 8, 2024
Merged

meta: support dir stream #5162

merged 13 commits into from
Oct 8, 2024

Conversation

jiefenghuang
Copy link
Contributor

@jiefenghuang jiefenghuang commented Sep 12, 2024

close #5099
ref #5110

  • redis: unchanged
  • sql: read full large directory by pagination will increase the total consumed time.

Signed-off-by: jiefenghuang <[email protected]>
Signed-off-by: jiefenghuang <[email protected]>
cmd/flags.go Outdated Show resolved Hide resolved
pkg/vfs/handle.go Outdated Show resolved Hide resolved
pkg/meta/tkv.go Outdated Show resolved Hide resolved
@jiefenghuang jiefenghuang marked this pull request as draft September 26, 2024 12:00
Signed-off-by: jiefenghuang <[email protected]>
Signed-off-by: jiefenghuang <[email protected]>
Signed-off-by: jiefenghuang <[email protected]>
Signed-off-by: jiefenghuang <[email protected]>
@jiefenghuang jiefenghuang marked this pull request as ready for review September 27, 2024 08:54
pkg/meta/base.go Outdated Show resolved Hide resolved
pkg/vfs/vfs.go Outdated Show resolved Hide resolved
Signed-off-by: jiefenghuang <[email protected]>
Signed-off-by: jiefenghuang <[email protected]>
Signed-off-by: jiefenghuang <[email protected]>
Signed-off-by: jiefenghuang <[email protected]>
Signed-off-by: jiefenghuang <[email protected]>
Signed-off-by: jiefenghuang <[email protected]>
pkg/meta/base.go Outdated Show resolved Hide resolved
pkg/meta/base.go Show resolved Hide resolved
pkg/meta/tkv.go Show resolved Hide resolved
Signed-off-by: jiefenghuang <[email protected]>
return err
}

s = s.Table(&edge{}).In("jfs_edge.id", ids).OrderBy("jfs_edge.name")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two queries for edge table can probably be merged as one.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And ordered by id should be a simpler way.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to sort by name, otherwise in some cases such as reverse order, the results may be incorrect

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SQL query is splited which allows it to directly hit the index tree and reduce the amount of querying.
Tried to merge statements in xorm, which would lose optimization effects.

A new pull request can be proposed for optimization.

@jiefenghuang jiefenghuang merged commit f3f4723 into main Oct 8, 2024
39 checks passed
@jiefenghuang jiefenghuang deleted the feat/streamdir branch October 8, 2024 02:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Suggestion] Change VFS readdir Function Behavior from Full Data Scan to Read-Ahead
4 participants