Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Copying large files to remote nodes in treemode is slow #557

Open
luxiaoyong opened this issue Mar 10, 2024 · 2 comments
Open

Copying large files to remote nodes in treemode is slow #557

luxiaoyong opened this issue Mar 10, 2024 · 2 comments

Comments

@luxiaoyong
Copy link

I use the treemode to copy a 136MB file to 142 remote nodes with the following command, which took 36 minutes:
clush -o-q -w "node1 node2" --copy test.tar --dest /tmp/
However, when I turn off the treemode, the same copy only took 20 seconds. Sometimes, copying large files may freeze (occasionally), but there are no error messages during the copying process. Could you please explain what protocol treemode uses for large file copying? Is it possible to optimize this feature?

@degremont
Copy link
Collaborator

This is a known limitation that the tree mode has sub-optimal performance. This is more a convenient feature for running commands or copying small files (think configuration files) on nodes that would not be reachable otherwise.
This is not a file transfer feature and does not compare to scp, rsync, etc...

We can keep this ticket open as a request to implement a faster transfer mode, but this is low priority for now.

@luxiaoyong
Copy link
Author

Thank you. I see. I used other methods to avoid this problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants