Add SigLIP

[NielsRogge]

What does this PR do?

This PR adds Google's new SigLIP model (CLIP with a better loss function). It's based on the Google Colab provided by the authors.

cc @patil-suraj feel free to take over this one

To do:

• add SiglipTokenizer (or use T5Tokenizer ? The vocab is defined here)
• add tests for the image processor, tokenizer and processor
• add fast tokenizer and enable fast tokenizer tests => skip fast tokenizer for now, see branch add_siglip_fast_tokenizer
• add loss function for training => won't do since various torch.distributed utilities would have to be incorporated
• important one: make sure that weights of SiglipVisionModel can be properly loaded without from_pretrained complaining
• make sure attention_mask is not passed for siglip checkpoints by updating model_input_names for checkpoints
• set split_special_tokens=True? => no but users can pass this flag
• transfer checkpoints, update organization name

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[ArthurZucker]

Not a fiinal review! Hope it helps

[NielsRogge]

@ArthurZucker I added 26590d2 for split_special_tokens=True which required me to overwrite some tests of tokenization_common.py. Could you have a look?

Also, this isn't supported by tokenizers yet right?

To me it feels a bit weird to have this behaviour by default to match the original implementation, since any original implementation won't ever keep special tokens.

[ArthurZucker]

Thanks for adding this new model!
Feel free to merge, the last test would be a nice to have, only thing to adress is the padding max length that you force in the pipeline

[ArthurZucker]

Last thing is this !

[VictorSanh]

thanks for adding this!

is there a reason why processor(text=["hello bonjour", "bonjour"], return_tensors="pt", padding=True) does not return any attention mask?

Perhaps it refers to make sure attention_mask is not passed for siglip checkpoints by updating model_input_names for checkpoints but i am not sure i understand why

>>> processor.tokenizer(["hello bonjour", "bonjour"], padding=True, return_attention_mask=True)
{'input_ids': [[14647, 10048, 20852, 1], [10048, 20852, 1, 1]], 'attention_mask': [[1, 1, 1, 1], [1, 1, 1, 0]]}
>>> processor(text=["hello bonjour", "bonjour"], padding=True, return_attention_mask=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __call__() got an unexpected keyword argument 'return_attention_mask'

it looks like return_attention_mask is not passed to the tokenizer in the call to the processor

[NielsRogge]

Hi @VictorSanh, SigLIP was trained without attention_mask (said differently, the text encoder attends to all tokens, including padding tokens!). Hence I explicitly had to set model_input_names only to "input_ids" for the checkpoints on the hub such that the model will internally attend to all tokens.

We still provide the possibility to create an attention_mask if you want padding tokens to be ignored, although predictions with the existing checkpoints will be pretty bad as that's not how those were trained.

Regarding the return_attention_mask argument not being passed to the tokenizer, indeed that's not supported yet. I'll add it as part of #28578

[VictorSanh]

got it, i didn't see the issue.
that's quite odd that attention mask was not used

[xenova]

@VictorSanh another thing to note (which tripped me up), is that you need to use padding='max_length'... otherwise, the output differs wildly (see here for more info).

[VictorSanh]

interesting, thanks for the info

these are rather odd behaviors (in comparison to what other tokenizers & models behave). do you think we can display that info somewhere? in the doc or the model card for instance.

[amyeroberts]

@VictorSanh Behaviour and doc examples were updated in #28578

[VictorSanh]

thank you!

[HugoLaurencon]

Hi, could someone explain why you chose to use Bicubic interpolation over Bilinear ones for the resizing of the images? In the official BigVision repo, I find bilinear methods but not bicubic ones.
https://github.com/google-research/big_vision/blob/main/big_vision/pp/ops_image.py

[amyeroberts]

Hi, could someone explain why you chose to use Bicubic interpolation over Bilinear ones for the resizing of the images? In the official BigVision repo, I find bilinear methods but not bicubic ones. https://github.com/google-research/big_vision/blob/main/big_vision/pp/ops_image.py

@NielsRogge good motivation to fill out #28180

[davidgxue]

@NielsRogge Quick question if you don't mind: why is self.num_positions = self.num_patches? CLIP and other vision transformers has +1 for num of positions. Is it because there's not CLASS embedding/token? I dug super deep and realized the original implementation from here google-research/vision_transformer#61 they add a CLASS embedding to keep it consistent as the generic Transformers architecture.

I thought well, maybe SigLIP is different. Maybe Google implemented it without the CLASS embedding this time. Hence why it doesn't have the +1. But from Google's repo (https://github.com/google-research/big_vision/blob/d0b297bbb8e073861d2d09ebd57ca46496357e36/big_vision/configs/proj/image_text/siglip_lit_coco.py#L81), this line has the pool type set to tok which if you go to their vit modeling file, adds the CLASS embedding still.

So... I am guessing it just so happens you didn't add the CLASS token/embedding in this implementation? But I am trying to figure out the reasoning behind it. Perhaps there's some code you followed that does that?
Validating my thoery: I also see you don't have self.class_embedding declared in the SiglipVisionEmbeddings class, whereas models like CLIP or VIT has it initialized for their embeddings class.

I am trying to work on #30579 for SigLIP, so trying to understand it better. If class embedding isn't added then the interpolate function would differ slightly. I think it maybe safe assume you are not adding the CLASS token just based on this implementation, but not 100% sure. Confirming to be safe

[NielsRogge]

I don't think SigLIP has a CLS token indeed.

[rootAvish]

@NielsRogge If you don't mind answering a question on this:

add loss function for training => won't do since various torch.distributed utilities would have to be incorporated

The docs say to use the SigLIP loss from the open_clip repository. Going over the code from open_clip I see that that it is indeed implemented using a lot of functionality from torch.distributed, but I didn't quite understand why this one can't be implemented without torch.distributed, the function in open_clip also allows for a world size of 1 right (which is equivalent to not using torch.distributed)?

[NielsRogge]

@rootAvish feel free to open a PR, I'm not sure it would be equivalent in that case (e.g. if you would then use the 🤗 Trainer API and run on multiple devices, are gradients synced in the same way?).