Does the 1.15 release support the new v3 ggml format?

[watcher60]

Sorry for the dumb question- does today’a 1.15.0 release bring support for the v3 model formats such as Manticore.

I believe it should but I’m having bad magic errors when attempting to load. I have updated the python package llama-cop-python) and been able to load the model that way successfully, just not with LocalAI

[mudler]

Please don't post any model which license is not permissive.

Manticore here works:

│ local-ai-79684c99d5-bslmr 8:22AM DBG Request received: {"model":"gpt-3.5-turbo","file":"","language":"","response_format":"","size":"","prompt":null,"instruction":"","input":null,"stop":null,"messages":[{"role":"system","content":"Yo │
│ local-ai-79684c99d5-bslmr 8:22AM DBG Parameter Config: &{OpenAIRequest:{Model:Manticore-13B.ggmlv3.q5_1.bin File: Language: ResponseFormat: Size: Prompt:<nil> Instruction: Input:<nil> Stop:<nil> Messages:[] Stream:false Echo:false To │
│ local-ai-79684c99d5-bslmr 8:22AM DBG Stream request received                                                                                                                                                                              │
│ local-ai-79684c99d5-bslmr 8:22AM DBG Template found, input modified to: ### Instruction: You are ChatGPT, a large language model trained by OpenAI. Follow the user's instructions carefully. Respond using markdown.                     │
│ local-ai-79684c99d5-bslmr how are you?                                                                                                                                                                                                    │
│ local-ai-79684c99d5-bslmr                                                                                                                                                                                                                 │
│ local-ai-79684c99d5-bslmr ### Assistant:                                                                                                                                                                                                  │
│ local-ai-79684c99d5-bslmr [10.42.0.67]:55288  200  -  POST     /v1/chat/completions                                                                                                                                                       │
│ local-ai-79684c99d5-bslmr 8:22AM DBG Loading model llama from Manticore-13B.ggmlv3.q5_1.bin                                                                                                                                               │
│ local-ai-79684c99d5-bslmr 8:22AM DBG Loading model in memory from file: /models/Manticore-13B.ggmlv3.q5_1.bin                                                                                                                             │
│ local-ai-79684c99d5-bslmr llama.cpp: loading model from /models/Manticore-13B.ggmlv3.q5_1.bin                                                                                                                                             │
│ local-ai-79684c99d5-bslmr 8:22AM DBG Sending chunk: {"object":"chat.completion.chunk","model":"gpt-3.5-turbo","choices":[{"delta":{"role":"assistant"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}              │
│ local-ai-79684c99d5-bslmr                                                                                                                                                                                                                 │
│ local-ai-79684c99d5-bslmr llama_model_load_internal: format     = ggjt v3 (latest)                                                                                                                                                        │
│ local-ai-79684c99d5-bslmr llama_model_load_internal: n_vocab    = 32000                                                                                                                                                                   │
│ local-ai-79684c99d5-bslmr llama_model_load_internal: n_ctx      = 1024                                                                                                                                                                    │
│ local-ai-79684c99d5-bslmr llama_model_load_internal: n_embd     = 5120                                                                                                                                                                    │
│ local-ai-79684c99d5-bslmr llama_model_load_internal: n_mult     = 256                                                                                                                                                                     │
│ local-ai-79684c99d5-bslmr llama_model_load_internal: n_head     = 40                                                                                                                                                                      │
│ local-ai-79684c99d5-bslmr llama_model_load_internal: n_layer    = 40                                                                                                                                                                      │
│ local-ai-79684c99d5-bslmr llama_model_load_internal: n_rot      = 128                                                                                                                                                                     │
│ local-ai-79684c99d5-bslmr llama_model_load_internal: ftype      = 9 (mostly Q5_1)                                                                                                                                                         │
│ local-ai-79684c99d5-bslmr llama_model_load_internal: n_ff       = 13824                                                                                                                                                                   │
│ local-ai-79684c99d5-bslmr llama_model_load_internal: n_parts    = 1                                                                                                                                                                       │
│ local-ai-79684c99d5-bslmr llama_model_load_internal: model size = 13B                                                                                                                                                                     │
│ local-ai-79684c99d5-bslmr llama_model_load_internal: ggml ctx size =    0.09 MB                                                                                                                                                           │
│ local-ai-79684c99d5-bslmr llama_model_load_internal: mem required  = 11359.05 MB (+ 1608.00 MB per state)   
│ local-ai-79684c99d5-bslmr 8:22AM DBG Sending chunk: {"object":"chat.completion.chunk","model":"gpt-3.5-turbo","choices":[{"delta":{"content":" Hello"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}              │

I've installed it with:

curl $LOCALAI/models/apply -H "Content-Type: application/json" -d '{
     "url": "github:go-skynet/model-gallery/manticore.yaml",
     "name": "gpt-3.5-turbo",
     "overrides": { "parameters": {"model": "Manticore-13B.ggmlv3.q5_1.bin" }, "f16": true },
     "files": [
        {
            "uri": "xxxxx/resolve/main/Manticore-13B.ggmlv3.q5_1.bin",
            "sha256": "7d2c76516bcfdedc0d6282e3c352e2423964989fc871e21b1922f0f1b8acc1db",
            "filename": "Manticore-13B.ggmlv3.q5_1.bin"
        }
     ]
   }'

[watcher60]

@mudler - apologies for posting a link (I was not aware of the licensing concern and duly noted going forward). Appreciate the response, for some reason I appear to be stuck on the v2 file format as I managed to get a copy from the other branch for the above which does work. I had done a fresh git pull, but I'll try completely a fresh in a new folder to see if that resolves things.

The issue is obviously local to me which is what my original query was as I didnt want to be spinning my wheels if it was not.
Thanks again