Incorrect Backend Selected

[apcameron]

When I select a model it does not seem to pickup the correct backend in this case.

How do I get t to associate the model with the correct backend?

In this case I am trying to use a model that is for use with starcoder.

Starting LocalAI using 4 threads, with models path: /home/andrew/Downloads/LocalAI/models

┌───────────────────────────────────────────────────┐
│ Fiber v2.46.0 │
│ http://127.0.0.1:8080 │
│ (bound on host 0.0.0.0 and port 8080) │
│ │
│ Handlers ............ 23 Processes ........... 1 │
│ Prefork ....... Disabled PID ............. 27409 │
└───────────────────────────────────────────────────┘

llama.cpp: loading model from /home/andrew/Downloads/LocalAI/models/gpt_bigcode-santacoder-ggml-q4_1.bin
error loading model: missing tok_embeddings.weight
llama_init_from_file: failed to load model
gptj_model_load: loading model from '/home/andrew/Downloads/LocalAI/models/gpt_bigcode-santacoder-ggml-q4_1.bin' - please wait ...
gptj_model_load: n_vocab = 49280
gptj_model_load: n_ctx = 2048
gptj_model_load: n_embd = 2048
gptj_model_load: n_head = 16
gptj_model_load: n_layer = 24
gptj_model_load: n_rot = 2003
gptj_model_load: f16 = 49280
gptj_model_load: invalid model file '/home/andrew/Downloads/LocalAI/models/gpt_bigcode-santacoder-ggml-q4_1.bin' (bad vocab size 1 != 49280)
GPT-J ERROR: failed to load model from /home/andrew/Downloads/LocalAI/models/gpt_bigcode-santacoder-ggml-q4_1.bingpt_neox_model_load: loading model from '/home/andrew/Downloads/LocalAI/models/gpt_bigcode-santacoder-ggml-q4_1.bin' - please wait ...
gpt_neox_model_load: n_vocab = 49280
gpt_neox_model_load: n_ctx = 2048
gpt_neox_model_load: n_embd = 2048
gpt_neox_model_load: n_head = 16
gpt_neox_model_load: n_layer = 24
gpt_neox_model_load: n_rot = 2003
gpt_neox_model_load: par_res = 49280
gpt_neox_model_load: ftype = 1
gpt_neox_model_load: qntvr = 0

[mudler]

In order to specify a backend you have to set backend in the model YAML configuration, see for example https://github.com/go-skynet/LocalAI/blob/2a11f16c0ffee07a128aeb92ad534650fcec161c/examples/rwkv/models/gpt-3.5-turbo.yaml#L9

The available backends are listed here https://localai.io/model-compatibility/index.html

[luoweb]

load gpt_bigcode-santacoder-ggml-q4_1.bin model failed

name: santacoder
# Default model parameters
parameters:
  # Relative to the models path
  model: gpt_bigcode-santacoder-ggml-q4_1.bin
backend: starcoder
context_size: 8192
threads: 4
debug: true
prompt_cache_all: false
template:
#   complete: completion
  chat: santacoder

┌───────────────────────────────────────────────────┐
│ Fiber v2.47.0 │
│ http://127.0.0.1:8080 │
│ (bound on host 0.0.0.0 and port 8080) │
│ │
│ Handlers ............ 32 Processes ........... 1 │
│ Prefork ....... Disabled PID ............. 59956 │

1:56PM DBG Template found, input modified to: <|system|> Below is a conversation between a human user and a helpful AI coding assistant. <|end|>
<|user|> how do i sort a list in python? <|end|>
<|assistant|>
1:56PM DBG Loading model starcoder from gpt_bigcode-santacoder-ggml-q4_1.bin
1:56PM DBG Loading model in memory from file: /Users/block/code/data/models/gpt_bigcode-santacoder-ggml-q4_1.bin
starcoder_model_load: loading model from '/Users/block/code/data/models/gpt_bigcode-santacoder-ggml-q4_1.bin'
starcoder_model_load: n_vocab = 49280
starcoder_model_load: n_ctx = 2048
starcoder_model_load: n_embd = 2048
starcoder_model_load: n_head = 16
starcoder_model_load: n_layer = 24
starcoder_model_load: ftype = 1003
starcoder_model_load: qntvr = 1
starcoder_model_load: ggml ctx size = 1626.91 MB
starcoder_model_load: memory size = 768.00 MB, n_mem = 49152
starcoder_model_load: unknown tensor '' in model file
starcoder_bootstrap: failed to load model from '/Users/block/code/data/models/gpt_bigcode-santacoder-ggml-q4_1.bin'
[127.0.0.1]:63403 500 - POST /v1/chat/completions

[apcameron]

Thanks.
Is there an example of a yaml file that could be used for starcoder?
Must the yaml files be placed in the LocalAI/models folder?
Is there a naming convention that needs to be used based on the model file?

It would be nice to have updated documentation with examples for the models that are not covered in the existing documentation.

[apcameron]

I think the documents at https://localai.io/advanced/index.html will help

[apcameron]

I found the documentation needed.