timm to pytorch conversion for vit model fix

[staghado]

This PR fixes this issue #26219 with timm to PyTorch conversion. It removes the need for hard coded values for model dims by using the attributes of the timm model without needing the model name.

It does the following things :

• Extract model dims from the timm model directly, no need for ifs
• Decides whether the converted model will be a classification model or only a feature extractor using the num_classes attribute of the timm model.
• In the case of a feature extractor only model : remove the pooling layers from the PyTorch model and compare the output to the last hidden state instead.

This works for a large number of models in the ViT family.

@ArthurZucker, @amyeroberts, @rwightman

[ArthurZucker]

Thanks for the cleanup, LGTM but I need a second look from @rwightman 🤗

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[rwightman]

Overall looking much better, should be more robust now.

A few things, there are a number of vit configurations supported in timm that are not, to my knowledge, supported in transformers. Should there be an attempt to detect? Thinking of some examples

• if fc_norm is present (norm after pooling)
• use of global average pooling in combination (or without) class token
• non-overlapping position and class token embedding
• CLIP style vit with norm_pre layer present
• SigLIP style vit with attn_pool layer present
• and soon, use of 'registers' via reg_token param
• use of layer scale in ViT model blocks

The git history is completely messed up!

[ArthurZucker]

Hey! Make sure to rebase to only have your changes! 😉

[staghado]

Hey! Make sure to rebase to only have your changes! 😉

I have reset the branch's history and left only my changes which fix the issue here.

[staghado]

Overall looking much better, should be more robust now.

A few things, there are a number of vit configurations supported in timm that are not, to my knowledge, supported in transformers. Should there be an attempt to detect? Thinking of some examples

• if fc_norm is present (norm after pooling)
• use of global average pooling in combination (or without) class token
• non-overlapping position and class token embedding
• CLIP style vit with norm_pre layer present
• SigLIP style vit with attn_pool layer present
• and soon, use of 'registers' via reg_token param
• use of layer scale in ViT model blocks

I have tried to add some checks before trying to convert the model from timm to huggingface.
Checks to be added :
1. non-overlapping position and class token embedding
2. use of 'registers' via reg_token param
3. check when a model has a convolution feature extractor like ResNet50

I have tested the script on the pre-trained ViTs and only the following give errors:

• vit_base_r50_s16_224.orig_in21k (contains a resnet block)
• vit_base_r50_s16_384.orig_in21k_ft_in1k (contains a resnet block)
• vit_small_r26_s32_224.augreg_in21k
• vit_small_r26_s32_224.augreg_in21k_ft_in1k
• vit_small_r26_s32_384.augreg_in21k_ft_in1k
• vit_tiny_r_s16_p8_224.augreg_in21k
• vit_tiny_r_s16_p8_224.augreg_in21k_ft_in1k
• vit_tiny_r_s16_p8_384.augreg_in21k_ft_in1k

[rwightman]

@staghado looking good, those hybrid resnet-vit models should be possible to catch (see if below) with a meaningful error .. other than looks ready to go

if not isinstance(model.patch_embed, timm.layers.PatchEmbed) ...

[rwightman]

looks good from the timm perspective

[amyeroberts]

Thanks for adding this and making the script more general!

Just a small question on an outstanding to-do. Otherwise LGTM!

[staghado]

@ArthurZucker

[ArthurZucker]

Thanks for improving this script! 🚀