Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minor parallelism improvements #18

Open
mcandre opened this issue May 10, 2021 · 0 comments
Open

Minor parallelism improvements #18

mcandre opened this issue May 10, 2021 · 0 comments

Comments

@mcandre
Copy link

mcandre commented May 10, 2021

Can we please provide a faster default for the number of parallel file transfers? I think a default setting of 4 concurrent file transfers would really improve out of the box performance.

  • 4 matches similar concurrency settings for FTP clients.
  • 4 integrates well with modern CPU core count configurations.
  • 4 has low risk of triggering transfer errors in Kubernetes.
  • 4 has low risk of triggering transfer errors in S3.

Also, I wonder if the value of 0 for the concurrency setting should be rejected by skbn. This is because full concurrency of all files at once, tends to trigger transfer cooldown errors. Because file trees with very many files, will basically create a denial of service attack as far as S3 sees.

@mcandre mcandre changed the title Default parallelism to full Default parallelism to the number of host CPU cores May 10, 2021
@mcandre mcandre changed the title Default parallelism to the number of host CPU cores Better parallelism default May 10, 2021
@mcandre mcandre changed the title Better parallelism default Minor parallelism improvements May 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant