conversion can fail at ControlNet on machines with 16GB of system memory

[ssube]

Converting models on a system with 16GB of system memory, without the onnx-fp16 or torch-fp16 optimizations, can lead to an out of memory error:

This appears to be related to the ControlNet conversion, which makes some sense, because that is effectively a copy of the UNet and the UNet is already the largest model. However, @HoopyFreud reports this was not happening on the previous conversion method, so it may also be related to #337

I have not documented the minimum specs yet, but would like to support 4c/16GB machines with 4GB of VRAM.

[ssube]

I've confirmed this on a 16GB laptop, where converting almost always fails with an Aborted or out of memory error.

Looking at the memory profiler, the usage during conversion can easily hit 15GB:

That doesn't leave much memory for the system and will typically fail on a machine with 16GB. According to the profiler, some 6GB of memory are being used by Torch and/or CUDA.

When converting diffusion models, the full pipeline is loaded, each model is exported to ONNX, then the ONNX models are reloaded in order to be optimized (single external tensor file, fp16, etc). Recent changes for ControlNet added some additional load_model calls before the UNet has been fully unloaded, which can load a second copy of the CNet/UNet and take 3-4GB of memory.

[ssube]

This is working better and I was able to convert the base models on a 16GB laptop, but it can still freeze during conversion, and converting multiple models in a row seems to make that more likely. Based on the current info, it seems to be getting stuck during the cnet conversion: there is a valid unet model and the cnet folder exists, but is empty.

[ssube]

The new optimum-based converter should fully unload the unet before converting the cnet, with no option to share them. That should reduce memory use to the minimum possible through that code path.