[DevTask] Better caching for control models

[huchenlei]

Currently the control model cache stores the PlugableModel object, which has caused some issue previously for multiple IP-Adapter usage (#2469). The PlugableModel object for ip-adapter contains extra information that should not be shared between units, which caused the last registered unit being applied multiple times. The temporary fix to this was to exclude IP-Adapter model from model cache. However, this causes IP-Adapter models being loaded from disk every time it is used, and this can be very costly for SDXL ip-adapters.

As requested in #2119 (comment), we should rework the caching mechanism to store the underlying model object, which the state_dict directly loads to instead of PlugableModel object.