Is it possible to train with two gpus on the same training?

[ivanced09]

I have two rtx 3060 12gb and I see that the initial options appear options related to multi-gpu, but I would not know how to configure them, I can do simultaneous entry in parallel without problems on windows 10.

[maxenko]

Would be nice to know a YES or a NO to this, so we can stop guessing.
I've asked the same thing yesterday, and can't find my question anymore.

[ivanced09]

The answer is "yes" but only on linux is possible the "distributed training", and tried to install ubuntu and run the gui but I had some problem the drivers related to cuda or CuDnn, I did not keep trying simply for lack of patience but I think it is something easy to solve.

[mrmeseeks23]

How can this be accomplished on a platform like 'runpod.io'. I've had success get things up and running but an 80GB A100 is maxed out whilst training 35 images and runs SUPER slow. Apparently, according to the terminal 71 of the 80 are allocated for Torch and the remaining '7' is for the software specifically. I am not sure if this is expected, or if its something that is inconsequential. I have tried many different approaches. I have checked that yes both GPU's are recognized BUT only one is acknowledged during initiation via './gui.sh --share --headless'. Also the other gpu just sits there during training runs while the other one carries all the weight.

Or maybe another type of GPU might run better (or so i've read elsewhere).

[okingjo]

yes, it can, I`m doing training on a 6-GPU server and it works well.
all you need to do is just do the accelerate config, select the multiple-GPU for training. The hg accelerate is good enough for getting start, so answer no to all the optimizer unless you know how.

[mrmeseeks23]

I figured out the acceleration thing i believe. Are you able to comment on the following:

1. Deepspeed, megatron (I think that's what its called), and one other i cannot recall the name of. They ask during init of the acceleration config.

2. A definitive answer on what TensorRT does during training for AI imaging? I am having a helluva time getting all of the components built for that. Seems like everytime I think I have it figured out, another component ends up being the wrong version.

3. Even with the accel config running I was still having training times over several hours for a lora. All of the GPU's were firing per the GPU use visualizer on runpod. So I am not certain I understand why it was not moving tremendously faster.

4. Do you create separate virtual environments for each application within one pod/storage volume? (ie one for kohya, one fo rSD-webui, etc.)? MAybe that's where I was running into problems as I was trying to onboard tensorflow specifically within the Virtual environment. When I was installing system wide it would go bye bye everytime I closed a pod session.

I would be willing to make a guide for these things but I have struggled to get tensorflow built, which is honestly one of the biggest questions I have.