Model loading speed optimization

[RyanJDick]

What does this PR do?

This PR moves an unchanging operation out of a loop for a speed benefit during model loading. It does not change the functional behaviour in any way.

Explanation

The following code was used to profile model loading:

import cProfile
from diffusers import UNet2DConditionModel

def main():
    with cProfile.Profile() as pr:
        unet = UNet2DConditionModel.from_pretrained("runwayml/stable-diffusion-v1-5", subfolder="unet")
        pr.dump_stats("unet_load.prof")

if __name__ == "__main__":
    main()

Before:
The resultant .prof file can be visualized with snakeviz. This revealed that a significant amount of time was being spent on redundant calls to inspect.signature(....):

After:
After the improvement in this PR, the time spent on calls to inspect.signature(....) was reduced from 0.0340s to 0.0001s:

Aside

You may be wondering: "Why does this guy care about such a small slice of the flamegraph?".
A: I've already optimized many of the slower model loading steps (via torch monkey-patches), to the point where this is non-negligible. (And, it seemed like an easy fix.)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests? No new tests necessary - there is no change to the expected behaviour.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[yiyixuxu]

looking great!
thank you for keep squeezing out more performance for us:)

cc @patrickvonplaten @sayakpaul

[sayakpaul]

Ouf! Thank YOU!

[patrickvonplaten]

Very nice!