When using xformers in Ubuntu, it couldn't find the forward function replacement (removing --xformers works)

[cian0]

libs installed:

xformers 0.0.14.dev0
torch 1.13.0

Stack trace:

Traceback (most recent call last): File "/home/ian/repositories/kohya_ss/train_db_fixed_v7.py", line 1609, in <module> train(args) File "/home/ian/repositories/kohya_ss/train_db_fixed_v7.py", line 1248, in train noise_pred = unet(noisy_latents, timesteps, encoder_hidden_states).sample File "/home/ian/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(*input, **kwargs) File "/home/ian/miniconda3/lib/python3.9/site-packages/accelerate/utils/operations.py", line 507, in __call__ return convert_to_fp32(self.model_forward(*args, **kwargs)) File "/home/ian/miniconda3/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast return func(*args, **kwargs) File "/home/ian/miniconda3/lib/python3.9/site-packages/diffusers/models/unet_2d_condition.py", line 296, in forward sample, res_samples = downsample_block( File "/home/ian/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(*input, **kwargs) File "/home/ian/miniconda3/lib/python3.9/site-packages/diffusers/models/unet_2d_blocks.py", line 563, in forward hidden_states = attn(hidden_states, context=encoder_hidden_states) File "/home/ian/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(*input, **kwargs) File "/home/ian/miniconda3/lib/python3.9/site-packages/diffusers/models/attention.py", line 169, in forward hidden_states = block(hidden_states, context=context) File "/home/ian/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(*input, **kwargs) File "/home/ian/miniconda3/lib/python3.9/site-packages/diffusers/models/attention.py", line 217, in forward hidden_states = self.attn1(self.norm1(hidden_states)) + hidden_states File "/home/ian/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(*input, **kwargs) File "/home/ian/repositories/kohya_ss/train_db_fixed_v7.py", line 1552, in forward_xformers return self.to_out(out) File "/home/ian/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl return forward_call(*input, **kwargs) File "/home/ian/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 244, in _forward_unimplemented raise NotImplementedError(f"Module [{type(self).__name__}] is missing the required \"forward\" function") NotImplementedError: Module [ModuleList] is missing the required "forward" function steps: 0%| | 0/2200 [00:00<?, ?it/s] Traceback (most recent call last): File "/home/ian/miniconda3/bin/accelerate", line 8, in <module> sys.exit(main()) File "/home/ian/miniconda3/lib/python3.9/site-packages/accelerate/commands/accelerate_cli.py", line 43, in main args.func(args) File "/home/ian/miniconda3/lib/python3.9/site-packages/accelerate/commands/launch.py", line 837, in launch_command simple_launcher(args) File "/home/ian/miniconda3/lib/python3.9/site-packages/accelerate/commands/launch.py", line 354, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)

[cian0]

Traceback (most recent call last):
File "/home/ian/repositories/kohya_ss/train_db_fixed_v7.py", line 1609, in
train(args)
File "/home/ian/repositories/kohya_ss/train_db_fixed_v7.py", line 1248, in train
noise_pred = unet(noisy_latents, timesteps, encoder_hidden_states).sample
File "/home/ian/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ian/miniconda3/lib/python3.9/site-packages/accelerate/utils/operations.py", line 507, in call
return convert_to_fp32(self.model_forward(*args, **kwargs))
File "/home/ian/miniconda3/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
return func(*args, **kwargs)
File "/home/ian/miniconda3/lib/python3.9/site-packages/diffusers/models/unet_2d_condition.py", line 296, in forward
sample, res_samples = downsample_block(
File "/home/ian/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ian/miniconda3/lib/python3.9/site-packages/diffusers/models/unet_2d_blocks.py", line 563, in forward
hidden_states = attn(hidden_states, context=encoder_hidden_states)
File "/home/ian/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ian/miniconda3/lib/python3.9/site-packages/diffusers/models/attention.py", line 169, in forward
hidden_states = block(hidden_states, context=context)
File "/home/ian/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ian/miniconda3/lib/python3.9/site-packages/diffusers/models/attention.py", line 217, in forward
hidden_states = self.attn1(self.norm1(hidden_states)) + hidden_states
File "/home/ian/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ian/repositories/kohya_ss/train_db_fixed_v7.py", line 1552, in forward_xformers
return self.to_out(out)
File "/home/ian/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ian/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 244, in _forward_unimplemented
raise NotImplementedError(f"Module [{type(self).name}] is missing the required "forward" function")
NotImplementedError: Module [ModuleList] is missing the required "forward" function
steps: 0%| | 0/2200 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/ian/miniconda3/bin/accelerate", line 8, in
sys.exit(main())
File "/home/ian/miniconda3/lib/python3.9/site-packages/accelerate/commands/accelerate_cli.py", line 43, in main
args.func(args)
File "/home/ian/miniconda3/lib/python3.9/site-packages/accelerate/commands/launch.py", line 837, in launch_command
simple_launcher(args)
File "/home/ian/miniconda3/lib/python3.9/site-packages/accelerate/commands/launch.py", line 354, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)

[sgsdxzy]

Same problem here, on Windows.

[bmaltais]

@cian0 can you try to install xformers for linux in your venv with:

pip install git+https://github.com/facebookresearch/xformers@1d31a3a#egg=xformers

Not sure if this is how you installed it. It will take about 20 minutes to compile everything for your specific card.

[bmaltais]

Same problem here, on Windows.

Did xformer installed without error on your system? Are you running python 3.10.6?

Try intalling it again with pip install -U -I --no-deps https://github.com/C43H66N12O12S2/stable-diffusion-webui/releases/download/f/xformers-0.0.14.dev0-cp310-cp310-win_amd64.whl and see if it report errors...

[sgsdxzy]

Same problem here, on Windows.

Did xformer installed without error on your system? Are you running python 3.10.6?

Try intalling it again with pip install -U -I --no-deps https://github.com/C43H66N12O12S2/stable-diffusion-webui/releases/download/f/xformers-0.0.14.dev0-cp310-cp310-win_amd64.whl and see if it report errors...

Yes, xformers works correctly for me in other repos, like stable-diffusion-webui.

[geocine]

Please use python 3.10 , I noticed your errors are all pointing to 3.9

[xpdd123]

I use python3.8.10 in ubuntu20.04， I fix the error by refer to stable diffusion，torch==1.13.1+cu117 and xformers==0.0.16rc425， I modify the ubuntu_setup.sh
`#!/bin/bash
echo installing tk
sudo apt install python3-tk
python3 -m venv venv
source venv/bin/activate
pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117

pip install --use-pep517 --upgrade -r requirements.txt
pip install xformers==0.0.16rc425

accelerate config

echo -e "setup finished! run \e[0;92m./gui.sh\e[0m to start"
`
and then re-execute the script，so I can train with xformers in ubuntu

[bmaltais]

On windows that particular version tend to lead to NaN when training... Watch out. If your loss goes to NaN it is probably the reason...