Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Falcon-11B support #1737

Open
robertBrnnn opened this issue Jul 4, 2024 · 0 comments
Open

Falcon-11B support #1737

robertBrnnn opened this issue Jul 4, 2024 · 0 comments

Comments

@robertBrnnn
Copy link

Hi
I converted Falcon-11b using ct2-transformers-converter but I get the following error when trying to use the model.

Ctranslate version: 4.1.0

Conversion command:

ct2-transformers-converter --model tiiuae/falcon-11b --output_dir falcon-11b-base-ct2 --quantization int8 --trust_remote_code

Using model:

>>> import ctranslate2
>>> from transformers import AutoTokenizer
>>> model = ctranslate2.Generator("falcon-11b-base-ct2", device='cpu')
>>> tokenizer = AutoTokenizer.from_pretrained("tiiuae/falcon-11b")
>>> outputs = model.generate_batch([tokenizer.convert_ids_to_tokens(tokenizer.encode("Falcon 11b is a new LLM"))], sampling_topk=10, max_length=200, include_prompt_in_result=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: axis 2 has dimension 6144 but expected 4352
>>> 

I assume this is due to the new model being unsupported, will falcon 11b be supported by CT2?
Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant