Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IO optimization for verify command #4

Open
taoky opened this issue Aug 27, 2024 · 0 comments
Open

IO optimization for verify command #4

taoky opened this issue Aug 27, 2024 · 0 comments

Comments

@taoky
Copy link
Owner

taoky commented Aug 27, 2024

It's reported that it takes ~7-8 days for a full verification on a HDD ZFS array, though it's much faster on SSD.
taoky added a commit that referenced this issue Aug 27, 2024
@taoky
iterdir() optimization
bc61598 
Related: #4
taoky added a commit that referenced this issue Aug 27, 2024
@taoky
Save a bit syscalls in check_and_update()
f4927fd 
Related: #4
taoky added a commit that referenced this issue Aug 27, 2024
@taoky
Only do one stat() for packages when sync-packages is on in verificat…
d64ea61 
…ion step 3

Related: #4
taoky added a commit that referenced this issue Aug 27, 2024
@taoky
Parallelize verification step 3 with threads
799a477 
Related: #4
taoky added a commit that referenced this issue Aug 27, 2024
@taoky
Optimize get_existing_hrefs for fast path (json file exists)
9e57eb6 
Related: #4
taoky added a commit that referenced this issue Aug 27, 2024
@taoky
Parallelize verification step 4
99fabce 
Related: #4
taoky added a commit that referenced this issue Sep 3, 2024
@taoky @luvletter2333
Add fast_readall to save some syscalls
6e4aa61 
Related: #4
Co-authored-by: luvletter2333 <[email protected]>
taoky added a commit that referenced this issue Sep 3, 2024
@taoky
Replace glob by fast_iterdir in verification step 4
a5dbd61 
To avoid stat() every files in package/.
Related: #4
@taoky taoky pinned this issue Sep 3, 2024
taoky added a commit that referenced this issue Sep 4, 2024
@taoky
Add multithreading to genlocal
99bd4d9 
Related: #4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@taoky