UnboundLocalError: local variable 'pipe' referenced before assignment

[noxiouscardiumdimidium]

│ /media/piguy/STABLE-DIFFUSION1/Github/kohya_ss/train_network.py:873 in <module>                  │
│                                                                                                  │
│   870 │   args = parser.parse_args()                                                             │
│   871 │   args = train_util.read_config_from_file(args, parser)                                  │
│   872 │                                                                                          │
│ ❱ 873 │   train(args)                                                                            │
│   874                                                                                            │
│                                                                                                  │
│ /media/piguy/STABLE-DIFFUSION1/Github/kohya_ss/train_network.py:168 in train                     │
│                                                                                                  │
│   165 │   weight_dtype, save_dtype = train_util.prepare_dtype(args)                              │
│   166 │                                                                                          │
│   167 │   # モデルを読み込む                                                                     │
│ ❱ 168 │   text_encoder, vae, unet, _ = train_util.load_target_model(args, weight_dtype, accele   │
│   169 │                                                                                          │
│   170 │   # モデルに xformers とか memory efficient attention を組み込む                         │
│   171 │   train_util.replace_unet_modules(unet, args.mem_eff_attn, args.xformers)                │
│                                                                                                  │
│ /media/piguy/STABLE-DIFFUSION1/Github/kohya_ss/library/train_util.py:3149 in load_target_model   │
│                                                                                                  │
│   3146 │   │   if pi == accelerator.state.local_process_index:                                   │
│   3147 │   │   │   print(f"loading model for process {accelerator.state.local_process_index}/{a  │
│   3148 │   │   │                                                                                 │
│ ❱ 3149 │   │   │   text_encoder, vae, unet, load_stable_diffusion_format = _load_target_model(   │
│   3150 │   │   │   │   args, weight_dtype, accelerator.device if args.lowram else "cpu"          │
│   3151 │   │   │   )                                                                             │
│   3152                                                                                           │
│                                                                                                  │
│ /media/piguy/STABLE-DIFFUSION1/Github/kohya_ss/library/train_util.py:3125 in _load_target_model  │
│                                                                                                  │
│   3122 │   │   │   print(                                                                        │
│   3123 │   │   │   │   f"model is not found as a file or in Hugging Face, perhaps file name is   │
│   3124 │   │   │   )                                                                             │
│ ❱ 3125 │   │   text_encoder = pipe.text_encoder                                                  │
│   3126 │   │   vae = pipe.vae                                                                    │
│   3127 │   │   unet = pipe.unet                                                                  │
│   3128 │   │   del pipe                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
UnboundLocalError: local variable 'pipe' referenced before assignment
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /media/piguy/STABLE-DIFFUSION1/Github/kohya_ss/venv/bin/accelerate:8 in <module>                 │
│                                                                                                  │
│   5 from accelerate.commands.accelerate_cli import main                                          │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(main())                                                                         │
│   9                                                                                              │
│                                                                                                  │
│ /media/piguy/STABLE-DIFFUSION1/Github/kohya_ss/venv/lib/python3.9/site-packages/accelerate/comma │
│ nds/accelerate_cli.py:45 in main                                                                 │
│                                                                                                  │
│   42 │   │   exit(1)                                                                             │
│   43 │                                                                                           │
│   44 │   # Run                                                                                   │
│ ❱ 45 │   args.func(args)                                                                         │
│   46                                                                                             │
│   47                                                                                             │
│   48 if __name__ == "__main__":                                                                  │
│                                                                                                  │
│ /media/piguy/STABLE-DIFFUSION1/Github/kohya_ss/venv/lib/python3.9/site-packages/accelerate/comma │
│ nds/launch.py:918 in launch_command                                                              │
│                                                                                                  │
│   915 │   elif defaults is not None and defaults.compute_environment == ComputeEnvironment.AMA   │
│   916 │   │   sagemaker_launcher(defaults, args)                                                 │
│   917 │   else:                                                                                  │
│ ❱ 918 │   │   simple_launcher(args)                                                              │
│   919                                                                                            │
│   920                                                                                            │
│   921 def main():                                                                                │
│                                                                                                  │
│ /media/piguy/STABLE-DIFFUSION1/Github/kohya_ss/venv/lib/python3.9/site-packages/accelerate/comma │
│ nds/launch.py:580 in simple_launcher                                                             │
│                                                                                                  │
│   577 │   process.wait()                                                                         │
│   578 │   if process.returncode != 0:                                                            │
│   579 │   │   if not args.quiet:                                                                 │
│ ❱ 580 │   │   │   raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)    │
│   581 │   │   else:                                                                              │
│   582 │   │   │   sys.exit(1)                                                                    │
│   583                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯

[bmaltais]

I can't reproduce this issue. What OS, what version are you using?

[noxiouscardiumdimidium]

I can't reproduce this issue. What OS, what version are you using?

amd-x86/64 MB
linux xubuntu23.*
rtx 3060 12gb
tesla m40 24gb
whatever the freshest commit was, as of about an hour before post - then i tried reset --hard to the previous 7 verified commits...

wiping and starting over was the only thing that worked. it bricked nvidia drivers and cuda, system wipe - i dont understand how. had to reinstall those. im STILL getting cuda errors for oobabooga saying:
"text-generation-webui-text-generation-webui-1 | RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW"

the only experimental thing i tried yesterday was finally getting around to testing your docker...

EVERYTHING was glorious until i tried to test the docker install yesterday... it went so bad it bricked one of my drives... i had sudo su root to move files to a backup, chmod/chown to give my group permissions, and format the drive.. it somehow mounted a clone of the drive with a "1" at the end and moved all my files into it... i had reinstall all AI related envs from scratch, and replace all my model/config/docker files, and rebuild every other docker container with docker compose build --no-cache

side note: oobaboogas simple method for filetree access; "Volumes" mapped to relative addresses, mapped to tags

[bmaltais]

It is possible the multiple GPU caused an issue with the latest accelerate version. I have returned to the previous release for Linux systems... It might help with the issue.

[noxiouscardiumdimidium]

It is possible the multiple GPU caused an issue with the latest accelerate version. I have returned to the previous release for Linux systems... It might help with the issue.

^^(fresh linux install, no docker, no extra steps - still cant config accellerate)^^^

confirmed! and i dont know how to fix, i need your gui on the m40... i do 10k+ batches in runs that last 200+ hours.... i need the 3060 as main and for SD and textgen DE... (you pretty much own my m40, that thing is smoking hot 24 hours a day because of your much-beloved software... )

I did some debugging trying multiple fresh installs to system - it crashes on the last step of accelerate config :
15:11:23-153266 WARNING Could not automatically configure accelerate. Please manually configure accelerate with the option in the menu or with: accelerate config. Exiting Python virtual environment. Setup finished! Run ./gui.sh to start.
if accelerate is fully implemented in this rev, i might be able to get dual-gpu up again... (for a while it wouldn't honor my settings for local dual-gpu with megatron-xl, deepspeed, full-parallel-share and manually configed nvme offloads - one day it it just started trying to put everything into the vram of the 3060 regardless of my config....)

[bmaltais]

Have you tried to manually configure accelerate with: accelerate config

You might have to call it in the venv with ./venv/bin/accelerate config

[noxiouscardiumdimidium]

./venv/bin/accelerate config

for some reason the linux install is doing the mac library error
Warning: LD_LIBRARY_PATH environment variable is not set. Certain functionalities may not work correctly. Please ensure that the required libraries are properly configured.

in and out of venv... i dont think the setup is calling venv... i can config accelerate for it, it installs tensorrt, i launch and cant find tensorrt error. getting library error, and both gpus individually configed and exported proper cuda archs and ids, and both together, configed in and out of venv - same errors. (* tried with and without

1. library path
2. called pipe before assigned
3. and all accellerate auto-configs fail
4. cant find tensorrt

[noxiouscardiumdimidium]

i wiped linux repos, cleared almost everything including python and visual studio...after rebuilding every env/venv/d
ocker on my machine, got the 3060 working (as long as m40 wasnt a visable device).... m40 still refused to work until i installed nvidia 515 server.... it works now. havent tried reconfiguering dual yet...

[noxiouscardiumdimidium]

@bmaltais

1. THE PIPE: i managed to reproduce it; it was because of how it remapped my drive name and mount point - i hadnt realized yet so all my folder names were wrong..
2. OTHER ENVs: everything was on the same drive; all those folder paths had a new physical mount point and a "1" added to logical address - .conda/miniconda environmentalist stores everything as relative... thats why i had to reinstall conda as a quick-fix...

[yyk630]

What is the speed of using a Tesla M40 graphics card?