OpenAI to HF: Add large-v3 to conversion script

[jstoone]

What does this PR do?

It adds the new large-v3 to the script which converts OpenAI model to HF.

The URL is copied from the PR releasing large-v3: openai/whisper#1761

I can't find any tests related to the conversion script, so I've gone ahead and skipped that. Let me know, if I've missed something.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sanchit-gandhi Do you know what has changed, since the model checkpoint seems to be a byte array instead of a directory?

[jstoone]

I've converted PR to draft, as I'm currently getting the following exception:

Traceback (most recent call last):
  File "/Users/jstoone/Sites/huggingface/transformers/src/transformers/models/whisper/convert_openai_to_hf.py", line 185, in <module>
    convert_openai_whisper_to_tfms(args.checkpoint_path, args.pytorch_dump_folder_path)
  File "/Users/jstoone/Sites/huggingface/transformers/src/transformers/models/whisper/convert_openai_to_hf.py", line 137, in convert_openai_whisper_to_tfms
    dimensions = original_checkpoint["dims"]
                 ~~~~~~~~~~~~~~~~~~~^^^^^^^^
TypeError: byte indices must be integers or slices, not str

[amyeroberts]

Hi @jstoone, thanks for opening this PR and contributing to the library!

There's some other PRs relating to this

• Fix Whisper Conversion Script: Correct decoder_attention_heads and _download function #26834
• [Whisper] Add large-v3 version support #27336
• [Whisper] Add conversion script for the tokenizer  #27338

I believe #26834 tackles the issue being seen here. As #27336 is a more complete update - handling the addition of cantonese - then that will be the PR to be merged in.

cc @ArthurZucker for reference.

[jstoone]

@amyeroberts That's amazing! The community moves so darn fast, it's amazing to see. I'll go ahead and close this one then. Thanks!

[gerrynjenny]

@amyeroberts

As #27336 is a more complete update - handling the addition of cantonese - then that will be the PR to be merged in.

Maybe I'm not understanding you, but the PR appears to only remove mention of Cantonese

[amyeroberts]

@gerrynjenny The diff in the open PR removes instructions in the docs mentioning that you have to specify 100 languages. This is because it's now inferred from the checkpoint when converting. The PR previously included logic which added cantonese as a language in the tokenizer mapping. However that has already been merged in with #27338