Add adapter support to CLIP

[calpt]

Closes #296. This PR includes the following changes:

• Integration of all adapter methods into CLIPModel, CLIPTextModel & CLIPVisionModel classes.
• Modify prefix tuning implementation to support different values for n_heads & input_size at different locations: This is required for CLIP text & vision encoder prefixes to be stored in the same PT pool.
• Introduction of InvertibleAdaptersWrapperMixin for model classes that support invertible adapters in a sub-module.

Install with:

pip install git+https://github.com/calpt/adapter-transformers.git@dev/clip

Tested the implementation for our current project.