[Feature]: Add embeddings api for Llama

[Harsha-Pulagam]

currently I load openai api server using the command
python3 -m vllm.entrypoints.openai.api_server --model Llama3-8B-Instruct --dtype auto --host 0.0.0.0 --port 8051 --gpu-memory-utilization 0.8 --enforce-eager
I want to try embedding using llama3 but. after loading i can see that embedding API is not loaded

I couldn't find any parm to enable embeddings.

Help me to enable embeddings API

[DarkLight1337]

This means that the model doesn't support embeddings in vLLM yet. You can edit this issue to request such implementation.

[Harsha-Pulagam]

Which models can generate embeddings in vllm endpoint?
I am using Llama 3

[DarkLight1337]

Currently, only Mistral is supported (#3734).

https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/__init__.py#L82

[Harsha-Pulagam]

Thanks for the reply.
I have edited the issue to request

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[DarkLight1337]

Closed as completed by #9806. Note that you need to set your architecture to LlamaModel for this to work.

[bernardgut]

@DarkLight1337

Note that you need to set your architecture to LlamaModel for this to work.

How do I do that ? Do you mean --config-format ?

[DarkLight1337]

@DarkLight1337

Note that you need to set your architecture to LlamaModel for this to work.

How do I do that ? Do you mean --config-format ?

No, I mean that the architectures field in HF config.json should contain LlamaModel instead of LlamaForCausalLM.

[bernardgut]

Wait I am confused I am running this in kubernetes is there no way to pass this as an argument at runtime ?

[DarkLight1337]

You can create a script to edit the config.json prior to running vLLM, if needed.

[DarkLight1337]

Alternatively, you can create a fork of the HF repo with the changes to config.json. Another way would be to open a PR to allow LlamaForCausalLM to be used directly as an embedding model.