Add adapter support to BEiT

[jannik-brinkmann]

Integrates adapters into BEiT model. I ran adapter_tests/test_beit.py without errors and trained a BEiT-Base model with "houlsby" adapters on EuroSAT dataset achieving 86 % accuracy after 20 epochs with LR 5e-4 (still improving about 1 % per epoch) without any hyperparameter tuning. For comparison: With full fine-tuning of BEiT-Base on EuroSAT I achieved 89 % accuracy after 10 epochs with LR 3e-5.

[jannik-brinkmann]

Tests initially failed, as adapter support for BEiT was implemented without accounting for latest changes to main branch (UniPELT).

I adjusted the implementation and re-trained adapters on the above described model achieving similar accuracy (88 % after 20 epochs).

[calpt]

Hey @jannik-brinkmann,
thank you very much for your contribution! The implementation looks very good so far, just pointing out some minor points from the contribution guide that are still missing:

Implementation:

• Add <model_type>AdapterModel to the ADAPTER_MODEL_MAPPING_NAMES mapping in src/transformers/adapters/models/auto.py

Testing:

• Append <model_type> to the list in check_adapters.py.

Documentation:

• [...] Finally, list the file in index.rst.
• Add a new row for the model in the model table of the overview page at adapter_docs/model_overview.md, listing all the methods implemented by the new model.

[jannik-brinkmann]

Hi @calpt, thank you for your review! Please find below my response to each of your comments:

Implementation:

• Add <model_type>AdapterModel to the ADAPTER_MODEL_MAPPING_NAMES mapping in src/transformers/adapters/models/auto.py -> The file src/transformers/adapters/models/auto.py does not exist. ADAPTER_MODEL_MAPPING_NAMES is located in src/transformers/adapters/models/auto/adapter_model.py, where I already added <model_type>AdapterModel (documentation seems to be outdated). Please correct me, if I missed something.

Testing:

• Append <model_type> to the list in check_adapters.py -> I added <model_type> to the list. I remember searching in tests_adapters/ for that file during implementation, but was not able to find it (and then assumed it may just be that the documentation is outdated here as well). I would suggest to change the documentation to "Append <model_type> to the list in utils/check_adapters.py" to make it easier for future collaborators.

Documentation:

• [...] Finally, list the file in index.rst -> I updated index.rst
• Add a new row for the model in the model table of the overview page at adapter_docs/model_overview.md, listing all the methods implemented by the new model -> I updated adapter_docs/model_overview.md

[calpt]

Thanks for the quick response.

The file src/transformers/adapters/models/auto.py does not exist. ADAPTER_MODEL_MAPPING_NAMES is located in src/transformers/adapters/models/auto/adapter_model.py, where I already added <model_type>AdapterModel (documentation seems to be outdated). Please correct me, if I missed something.

You're absolutly correct, my apologies. Will update this in our docs.

I added <model_type> to the list. I remember searching in tests_adapters/ for that file during implementation, but was not able to find it (and then assumed it may just be that the documentation is outdated here as well). I would suggest to change the documentation to "Append <model_type> to the list in utils/check_adapters.py" to make it easier for future collaborators.

Thanks for the feedback, will make this more clear.

[calpt]

Thanks again for your contribution @jannik-brinkmann! We'd be happy to see any adapters you train with BEiT uploaded to the HuggingFace Hub. :)