About memory usage

[ChiaYen-Kan]

i have problem is about memory usage

i use second pass and hires fix to upscale image in txt2img
but even all prompt and the setting of second pass is same, sometime is success, sometime is "out of memory"
even after success generate image can't run again with same setting, second time is always "out of memory",
i need restart program to reset everything for run again txt2img

anyone have any good way to avoid "out of memory"?

Platform:
cpu: AMD64 Family 25 Model 80 Stepping 0, AuthenticAMD (amd ryzen 5 5600g without other gpu)
system: Windows
release: Windows-10-10.0.19045-SP0
python: 3.10.11
system memory is 48g, assign 16g for internal GPU, use directml

Backend: Original
Model: SD 1.5

the log is direct run second time without reset program
first time is succeeded, second time is "out of memory"

Using VENV: C:\AI\automatic-new\venv
09:05:02-455441 INFO     Starting SD.Next
09:05:02-459442 INFO     Python 3.10.11 on Windows
09:05:02-571466 INFO     Version: app=sd.next updated=2023-09-20 hash=89ba8e3c
                         url=https://github.com/vladmandic/automatic.git/tree/master
09:05:03-105744 INFO     Platform: arch=AMD64 cpu=AMD64 Family 25 Model 80 Stepping 0, AuthenticAMD system=Windows
                         release=Windows-10-10.0.19045-SP0 python=3.10.11
09:05:03-109746 INFO     Using DirectML Backend
09:05:03-262780 INFO     Extensions: disabled=[]
09:05:03-264781 INFO     Extensions: enabled=['a1111-sd-webui-lycoris', 'clip-interrogator-ext', 'LDSR', 'Lora',
                         'multidiffusion-upscaler-for-automatic1111', 'ScuNET', 'sd-extension-system-info',
                         'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'stable-diffusion-webui-images-browser',
                         'stable-diffusion-webui-rembg', 'SwinIR'] extensions-builtin
09:05:03-277783 INFO     Extensions: enabled=['Stable-Diffusion-Webui-Civitai-Helper',
                         'ultimate-upscale-for-automatic1111'] extensions
09:05:03-281784 INFO     No changes detected: Quick launch active
09:05:03-283784 INFO     Verifying requirements
09:05:03-303789 INFO     Verifying packages
09:05:03-306789 INFO     Extensions: disabled=[]
09:05:03-307789 INFO     Extensions: enabled=['a1111-sd-webui-lycoris', 'clip-interrogator-ext', 'LDSR', 'Lora',
                         'multidiffusion-upscaler-for-automatic1111', 'ScuNET', 'sd-extension-system-info',
                         'sd-webui-agent-scheduler', 'sd-webui-controlnet', 'stable-diffusion-webui-images-browser',
                         'stable-diffusion-webui-rembg', 'SwinIR'] extensions-builtin
09:05:03-320793 INFO     Extensions: enabled=['Stable-Diffusion-Webui-Civitai-Helper',
                         'ultimate-upscale-for-automatic1111'] extensions
09:05:03-332795 INFO     Extension preload: {'extensions-builtin': 0.01, 'extensions': 0.0}
09:05:03-334796 INFO     Command line args: ['--use-directml', '--models-dir', 'C:/AI/Models', '--medvram']
                         medvram=True models_dir=C:/AI/Models use_directml=True
09:05:11-752882 INFO     Engine: backend=Backend.ORIGINAL compute=directml mode=no_grad device=privateuseone:0
09:05:11-757883 INFO     Device:
09:05:12-480225 INFO     Using models path: C:/AI/Models
09:05:12-486951 INFO     Available VAEs: C:/AI/Models\VAE items=4
09:05:12-495944 INFO     Available models: C:/AI/Models\Stable-diffusion items=7 time=0.01s
09:05:16-093657 INFO     Extension: script='extensions-builtin\sd-webui-controlnet\scripts\controlnet.py' Warning:
                         ControlNet failed to load SGM - will use LDM instead.
09:05:16-337838 INFO     Extension: script='extensions-builtin\sd-webui-controlnet\scripts\controlnet.py' ControlNet
                         preprocessor location:
                         C:\AI\automatic-new\extensions-builtin\sd-webui-controlnet\annotator\downloads
09:05:16-348831 INFO     Extension: script='extensions-builtin\sd-webui-controlnet\scripts\hook.py' Warning: ControlNet
                         failed to load SGM - will use LDM instead.
Civitai Helper: Get Custom Model Folder
09:05:17-300638 INFO     Extensions time: 4.76s { a1111-sd-webui-lycoris=2.42s clip-interrogator-ext=0.05s Lora=0.18s
                         sd-webui-agent-scheduler=0.38s sd-webui-controlnet=0.67s
                         stable-diffusion-webui-images-browser=0.11s stable-diffusion-webui-rembg=0.66s
                         Stable-Diffusion-Webui-Civitai-Helper=0.09s }
09:05:17-360436 INFO     Loading UI theme: name=black-teal style=Auto
09:05:17-753524 INFO     Themes: builtin=6 default=5 external=48
09:05:19-933358 INFO     Local URL: http://127.0.0.1:7860/
09:05:19-935359 INFO     Initializing middleware
09:05:20-125401 INFO     [AgentScheduler] Task queue is empty
09:05:20-127769 INFO     [AgentScheduler] Registering APIs
Loading weights: C:\AI\Models\Stable-diffusion\lastpiececore_a0751+hassakuHentaiModel_v13.safetensors ----- 0.0… -:--: Loading weights: C:\AI\Models\Stable-diffusion\lastpiececore_a0751+hassakuHentaiModel_v13.safetensors ----- 0.0… -:--: Loading weights: C:\AI\Models\Stable-diffusion\lastpiececore_a0751+hassakuHentaiModel_v13.safetensors ----- 0.0… -:--: …
                                                                                                            GB
09:05:20-647544 INFO     Torch override dtype: no-half set
09:05:20-648544 INFO     Torch override VAE dtype: no-half set
09:05:20-650544 INFO     Setting Torch parameters: device=privateuseone:0 dtype=torch.float32 vae=torch.float32
                         unet=torch.float32 context=no_grad fp16=True bf16=False
09:05:21-387709 INFO     LDM: LatentDiffusion: Running in eps-prediction mode
09:05:21-389947 INFO     LDM: DiffusionWrapper has 859.52 M params.
Loading weights: C:/AI/Models\VAE\vae-ft-mse-840000-ema-pruned.safetensors ----------------------- 0.0/334.6 MB -:--:--
09:05:22-038091 INFO     Cross-attention: optimization=Sub-quadratic options=[]
09:05:22-050094 INFO     Loaded embeddings: loaded=3 skipped=0
09:05:22-060096 INFO     Model loaded in 1.59s { load=0.17s create=0.73s apply=0.50s vae=0.14s }
09:05:22-429179 INFO     Model load finished: {'ram': {'used': 7.15, 'total': 31.87}, 'gpu': {'used': 0.01, 'total':
                         0.01}, 'retries': 'DirectMLDevice', 'oom': 0} cached=0
09:05:22-497195 INFO     Startup time: 19.13s { torch=6.23s gradio=0.69s diffusers=0.89s libraries=1.30s
                         extensions=4.76s onchange=0.05s ui-txt2img=0.17s ui-img2img=0.12s ui-settings=0.09s
                         ui-extensions=1.84s ui-defaults=0.07s launch=0.22s api=0.07s app-started=0.25s
                         checkpoint=2.24s }
09:05:57-246784 INFO     Themes: builtin=6 default=5 external=48
09:07:21-940745 INFO     Loaded VAE-approx model: C:\AI\automatic-new\models\VAE-approx\model.pt
Loading weights: C:\AI\Models\Lora\bunnygirl.safetensors ---------------------------------------- 0.0/37.9 MB -:--:--
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [03:49<00:00,  7.64s/it]
09:11:23-775662 INFO     Model loaded: type=R-ESRGAN model=R-ESRGAN 4x+ Anime6B
        Tile 1/15
        Tile 2/15
        Tile 3/15
        Tile 4/15
        Tile 5/15
        Tile 6/15
        Tile 7/15
        Tile 8/15
        Tile 9/15
        Tile 10/15
        Tile 11/15
        Tile 12/15
        Tile 13/15
        Tile 14/15
        Tile 15/15
09:11:30-324233 INFO     GPU high memory utilization: 93% {'ram': {'used': 22.98, 'total': 31.87}, 'gpu': {'used':
                         15.91, 'total': 30.39}, 'retries': 'DirectMLDevice', 'oom': 0}
100%|██████████████████████████████████████████████████████████████████████████████████| 35/35 [17:02<00:00, 29.21s/it]
09:28:54-608517 INFO     Processed: images=1 time=1292.67s its=0.02 memory={'ram': {'used': 15.58, 'total': 31.87},
                         'gpu': {'used': 15.91, 'total': 30.39}, 'retries': 'DirectMLDevice', 'oom': 0}
100%|██████████████████████████████████████████████████████████████████████████████████| 30/30 [03:46<00:00,  7.56s/it]
09:39:42-460055 INFO     Model loaded: type=R-ESRGAN model=R-ESRGAN 4x+ Anime6B
        Tile 1/15
        Tile 2/15
        Tile 3/15
        Tile 4/15
        Tile 5/15
        Tile 6/15
        Tile 7/15
        Tile 8/15
        Tile 9/15
        Tile 10/15
        Tile 11/15
        Tile 12/15
        Tile 13/15
        Tile 14/15
        Tile 15/15
09:39:49-004377 INFO     GPU high memory utilization: 93% {'ram': {'used': 13.85, 'total': 31.87}, 'gpu': {'used':
                         15.91, 'total': 30.39}, 'retries': 'DirectMLDevice', 'oom': 0}
100%|██████████████████████████████████████████████████████████████████████████████████| 35/35 [16:58<00:00, 29.09s/it]
09:57:03-082589 ERROR    Exception: Could not allocate tensor with 452984832 bytes. There is not enough GPU video
                         memory available!
09:57:03-087590 ERROR    Arguments: args=('task(00e3hixoy41r5hc)', '((solo)), 1boy, (kawaii:1.2), cute, high
                         resolution, (incredibly absurdres:1.1), (hires.fix:1.3), anime visual, extremely detailed CG
                         unity 8k wallpaper, (flat chest:1.4), lying on side, detailed beautiful face, detailed
                         beautiful eyes, beautiful side ponytail, wavy hair, braided bangs, (blonde hair:1.5),
                         (beautiful side ponytail), light smile, blush, detailed room, (sky blue bunny suit),
                         (bunnygirl:1.4), (white bunny ear), white lace stockings, (shoes:1.3), (hard light:1.2),
                         <lora:bunnygirl:0.5>', '(easynegative), (negative_hand-neg:1.5), (badhandv4), (worst quality,
                         low quality:1.4), (nsfw:2.0), (nude:2.0), (deformity, bad legs, error legs, bad feet,
                         malformed limbs, extra limbs, mutation, bad anatomy, disfigured), jacket, logo, watermark,
                         signature', [], 30, 6, 6, True, False, False, 1, 1, 7, 6, 0.7, 2, 3701548916.0, -1.0, 0, 0, 0,
                         512, 768, True, 0.6, 1.5, 'R-ESRGAN 4x+ Anime6B', True, 35, 0, 0, 5, 0.8, 'high resolution,
                         (incredibly absurdres:1.1), (hires.fix:1.3), anime visual, extremely detailed CG unity 8k
                         wallpaper', '(easynegative), (negative_hand-neg:1.5), (badhandv4), (bad hand, fused fingers,
                         mutated hands, missing fingers, cloned), ((((logo, watermark, signature))))', ['VAE:
                         vae-ft-mse-840000-ema-pruned'], 0, False, False, 'positive', 'comma', 0, False, False, '', 0,
                         '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False, None, None, False, None,
                         None, False, None, None, False, 50, False, 'MultiDiffusion', False, True, 1024, 1024, 96, 96,
                         48, 4, 'None', 2, False, 10, 1, 1, 64, False, False, False, False, False, 0.4, 0.4, 0.2, 0.2,
                         '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0,
                         False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '',
                         'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False,
                         0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '',
                         'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False,
                         512, 64, True, True, True, False, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit
                         object at 0x000002CD87E05CC0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit
                         object at 0x000002CD87E268F0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit
                         object at 0x000002CD8AB2E020>) kwargs={}
09:57:05-033704 ERROR    gradio call: RuntimeError
┌───────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────┐
│ C:\AI\automatic-new\modules\call_queue.py:34 in f                                                                   │
│                                                                                                                     │
│   33 │   │   │   try:                                                                                               │
│ > 34 │   │   │   │   res = func(*args, **kwargs)                                                                    │
│   35 │   │   │   │   progress.record_results(id_task, res)                                                          │
│                                                                                                                     │
│ C:\AI\automatic-new\modules\txt2img.py:66 in txt2img                                                                │
│                                                                                                                     │
│   65 │   if processed is None:                                                                                      │
│ > 66 │   │   processed = processing.process_images(p)                                                               │
│   67 │   p.close()                                                                                                  │
│                                                                                                                     │
│ C:\AI\automatic-new\modules\processing.py:626 in process_images                                                     │
│                                                                                                                     │
│    625 │   │   else:                                                                                                │
│ >  626 │   │   │   res = process_images_inner(p)                                                                    │
│    627 │   finally:                                                                                                 │
│                                                                                                                     │
│ C:\AI\automatic-new\extensions-builtin\sd-webui-controlnet\scripts\batch_hijack.py:42 in                            │
│ processing_process_images_hijack                                                                                    │
│                                                                                                                     │
│    41 │   │   │   # we are not in batch mode, fallback to original function                                         │
│ >  42 │   │   │   return getattr(processing, '__controlnet_original_process_images_inner')(p,                       │
│    43                                                                                                               │
│                                                                                                                     │
│ C:\AI\automatic-new\modules\processing.py:764 in process_images_inner                                               │
│                                                                                                                     │
│    763 │   │   │   │   │   samples_ddim = p.sample(conditioning=c, unconditional_conditioning=u                     │
│ >  764 │   │   │   │   x_samples_ddim = [decode_first_stage(p.sd_model, samples_ddim[i:i+1].to(                     │
│    765 │   │   │   │   try:                                                                                         │
│                                                                                                                     │
│                                               ... 6 frames hidden ...                                               │
│                                                                                                                     │
│ C:\AI\automatic-new\modules\lowvram.py:52 in first_stage_model_decode_wrap                                          │
│                                                                                                                     │
│   51 │   │   send_me_to_gpu(first_stage_model, None)                                                                │
│ > 52 │   │   return first_stage_model_decode(z)                                                                     │
│   53                                                                                                                │
│                                                                                                                     │
│ C:\AI\automatic-new\repositories\stable-diffusion-stability-ai\ldm\models\autoencoder.py:90 in decode               │
│                                                                                                                     │
│    89 │   │   z = self.post_quant_conv(z)                                                                           │
│ >  90 │   │   dec = self.decoder(z)                                                                                 │
│    91 │   │   return dec                                                                                            │
│                                                                                                                     │
│ C:\AI\automatic-new\venv\lib\site-packages\torch\nn\modules\module.py:1501 in _call_impl                            │
│                                                                                                                     │
│   1500 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                                      │
│ > 1501 │   │   │   return forward_call(*args, **kwargs)                                                             │
│   1502 │   │   # Do not call functions when jit is used                                                             │
│                                                                                                                     │
│ C:\AI\automatic-new\repositories\stable-diffusion-stability-ai\ldm\modules\diffusionmodules\model.py:648 in forward │
│                                                                                                                     │
│   647 │   │   h = self.norm_out(h)                                                                                  │
│ > 648 │   │   h = nonlinearity(h)                                                                                   │
│   649 │   │   h = self.conv_out(h)                                                                                  │
│                                                                                                                     │
│ C:\AI\automatic-new\venv\lib\site-packages\torch\nn\functional.py:2059 in silu                                      │
│                                                                                                                     │
│   2058 │   │   return torch._C._nn.silu_(input)                                                                     │
│ > 2059 │   return torch._C._nn.silu(input)                                                                          │
│   2060                                                                                                              │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
RuntimeError: Could not allocate tensor with 452984832 bytes. There is not enough GPU video memory available!

[zethfox]

Interesting I just made a comment about this on the discord last night.
I'll paste it here as it sounds like the same exact issue I noticed...

i dont know what setting i have out of whack but im making a single image, 512x816 x2 upscale...and the gen info reports this Time: 15.96s | GPU active 18566 MB reserved 18566 | used 20342 MB free 4223 MB total 24564 MB....did it really just use 20gigs of my 24 to upscale a single 512x816 x2?.

which is any img past the first...the first one has its own issue....Time: 42.18s | GPU active 10210 MB reserved 3244 | used 4916 MB free 19648 MB total 24564 MB ...takes 42 secs...but that could be loading the model on first run...but notice atleast the vram usage isnt off the wall crazy like subsequent gens

batch size 6....Time: 3m 53.29s | GPU active 49398 MB reserved 3424 | used 5124 MB free 19440 MB total 24564 MB | retries 0 oom 0...clearly it dumped it to ram and used 49 gigs to make 6 imgs...

For comparison: auto1111 -- batch size 8 ..every setting identical..
Time taken: 1 min. 40.0 sec.
A: 9.10 GB, R: 12.39 GB, Sys: 15.1/23.9883 GB (62.8%)

My guess is that it appears to be reserving VRAM then not using what it reserved...I could be completely wrong.

Figured I would add this here for info....

My specs are i13 , rtx 4090, 128g ddr6...windows 11..

[vladmandic]

backend? model type?

[zethfox]

Original backend, 1.5sd model, 1.5sd base vae.
Additional info: using any sampler, using any NONLATENT upscaler (superyandere, superscale, ect) with force hires fix option checked.

Further info: Even with --safe, The problem persists. SD 1.5 models are 2gig pruned safetensors format.

[ChiaYen-Kan]

i have another high memory usage case but is system memory

in my system if change lora weight or reload lora model (remove lora prompt and re-add it in next loop) is take high system memory usage
when over some limit system is freeze 2~3 minute then python is direct crash
no exception, no log, just crash and i must restart program

so every time when i need change lora weight i need restart program first to avoid python crash

[vladmandic]

@ChiaYen-Kan very different scenario - its hard to bundle multiple problems in a single discussion thread.

[vladmandic]

@zethfox
using 100% default configuration, model=sd15, backend=original, size=512x816, upscaler=esrgan 4x ultrasharp, upscale=2, force hires, batch-size=8

first run:

10:13:59-620924 INFO Processed: images=6 time=132.82s its=0.88 memory={'ram': {'used': 5.7, 'total': 27.41}, 'gpu': {'used': 11.86, 'total': 22.49}, 'retries': 0, 'oom': 0}

second run:

10:17:42-241823 INFO Processed: images=6 time=133.63s its=0.84 memory={'ram': {'used': 5.7, 'total': 27.41}, 'gpu': {'used': 11.87, 'total': 22.49}, 'retries': 0, 'oom': 0}

i think its safe to say i cannot reproduce.

[vladmandic]

btw, don't do html fragments to post logs, simply add triple tick before and after

[vladmandic]

and this looks like a massive memory leak, haven't seen anything that big in ages. can you repeat that same run but with --debug flag and create an issue, this is going to get lost in discussions.

[zethfoxster]

ok np, didnt know how to break it out of that without losing the formatting on it...

[zethfoxster]

i sure can, ill do it now and list it in the bugs section

[vladmandic]

git uses markdown for formatting, its very efficient, i use it for my every-day note taking.

[vladmandic]

@ChiaYen-Kan regarding your first issue, not sure if i can help, the simple fact is that torch-directml is bad with memory (it forces everything in fp32 as it does not have proper fp16 implementation for all operations and autocast is non-functional). i've been waiting for update to torch-directml for a while, but there is nothing for months.

[ChiaYen-Kan]

thank you
for now i can use restart way to avoid "out of memory"
just a little annoying and take more time to generate image