The documentation and behavior of LocalAI are causing me significant confusion.

[wencan]

As a first-time user, I find the documentation and behavior of LocalAI to be very perplexing. I am unsure whether I have properly set up and running LocalAI.

I followed the instructions to launch LocalAI using the provided command:

docker run --rm -it -v /home/wencan/Projects/models/Qwen/Qwen1.5-7B-Chat-GGUF:/models --network host  localai/localai:latest-aio-cpu --models-path /models --context-size 512 --threads 4

The models directory contains only a single gguf file.

Upon launching LocalAI, it randomly downloads a model. Since I am located in mainland China, this download is bound to fail.

LocalAI then displays the listening address. I accessed the endpoint page, but the content is completely unrelated to my needs. The specified model is not found on the page, and the chat function is also unusable.

Using curl http://localhost:8080/v1/models, I obtained the following output:

{"object":"list","data":[{"id":"gpt-4","object":"model"},{"id":"gpt-4-vision-preview","object":"model"},{"id":"jina-reranker-v1-base-en","object":"model"},{"id":"stablediffusion","object":"model"},{"id":"text-embedding-ada-002","object":"model"},{"id":"tts-1","object":"model"},{"id":"whisper-1","object":"model"},{"id":"qwen1_5-7b-chat-q4_k_m.gguf","object":"model"}]}

I am unsure of the origin of the listed models except for the last one.

Finally, I attempted to initiate a conversation with the model using curl:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "qwen1_5-7b-chat-q4_k_m.gguf",
  "messages": [{"role": "user", "content": "Say this is a test!"}],
  "temperature": 0.7
}'

Regardless of the prompt or query, the response remains consistently the same:

{"created":1716421390,"object":"chat.completion","id":"68fa6cc2-7545-44e5-a0be-e78a96d2e313","model":"qwen1_5-7b-chat-q4_k_m.gguf","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"\u003c|im_end|\u003e"}}],"usage":{"prompt_tokens":6,"completion_tokens":1,"total_tokens":7}}

[mudler]

@wencan you are using the All-In-One images that comes with pre-configured models. If you don't need pre-configured models, use the standard images: https://localai.io/basics/container/

[wencan]

@mudler
Two issues found relevant to this discussion:

When I run the Docker image localai/localai:latest, which has the hash 47933b52bde3, it first outputs "===> LocalAI All-in-One (AIO) container starting..." and then starts downloading models. This indicates that it is an AIO image, not a standard image as stated in the documentation.

When I mount a local directory to an AIO container, the container automatically creates some files in that directory. When I next mount this directory to a standard container, the standard container will continue the AIO container's behavior. Unless I manually delete the files automatically created by the AIO container.