You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Thank you for your great work.
I'm trying to train a RoBERTa-based VLM model on my own dataset. I plan to use your pre-trained vokenizer provided here. But, in my understanding, your pre-trained vokenizer or cross-modal matching model is trained based on bert-base-uncased tokenizer and thus, I need revokenization to obtain the vokenization result, which will be used in the RoBERTa-based VLM training.
While I found the code revokenize_corpus_mp.py, it seems that Revokenizer is not used there (it's imported though).
Could you update the code accordingly? or could you give me some instruction for how to use the Revokenizer class in the code?
Also, I would appreciate it if you could provide a sample bash script like mpvokenize_wiki.bash to apply revokenization to a given dataset.
Thanks.
The text was updated successfully, but these errors were encountered:
Hi,
Thank you for your great work.
I'm trying to train a RoBERTa-based VLM model on my own dataset. I plan to use your pre-trained vokenizer provided here. But, in my understanding, your pre-trained vokenizer or cross-modal matching model is trained based on
bert-base-uncased
tokenizer and thus, I need revokenization to obtain the vokenization result, which will be used in the RoBERTa-based VLM training.While I found the code revokenize_corpus_mp.py, it seems that Revokenizer is not used there (it's imported though).
Could you update the code accordingly? or could you give me some instruction for how to use the Revokenizer class in the code?
Also, I would appreciate it if you could provide a sample bash script like mpvokenize_wiki.bash to apply revokenization to a given dataset.
Thanks.
The text was updated successfully, but these errors were encountered: