Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

can not distributed training #46

Open
ALR-alr opened this issue Oct 23, 2024 · 0 comments
Open

can not distributed training #46

ALR-alr opened this issue Oct 23, 2024 · 0 comments

Comments

@ALR-alr
Copy link

ALR-alr commented Oct 23, 2024

once I run the training with the following command:
CUDA_VISIBLE_DEVICES=0,1 randport=$(shuf -i8000-9999 -n1) # Generate a random port number python -u main.py \ --dist-url "tcp://127.0.0.1:${randport}" --dist-backend 'nccl' \ --multiprocessing-distributed --world-size 1 --rank 0 \ --dataset=cc3m --val-dataset=cc3m \ --exp-name='gill_exp' --image-dir='data/' --log-base-dir='runs/' \ --opt-version='/opt-6.7b' \ --visual-model /checkpoints/clip-vit-large-patch14 \ --precision='bf16' --print-freq=100
the server will be reconnected and the process will disconnect with the following error message:

torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with signal SIGKILL

UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown

warnings.warn('resource_tracker: There appear to be %d '

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant