Falcon 40B. Is any way to run with LocalAI? #1368

netandreus · 2023-11-28T07:45:47Z

netandreus
Nov 28, 2023

Is there any way to run Falcon 40B model with LocalAI?
I'm trying this models:

and this backends:

without mentioned (default)
backend: falcon
backjend: falcon-ggml

without any success.

My model config.yml file:

context_size: 2000
f16: true
gpu_layers: 1
name: wizardlm-uncensored-falcon-40b
parameters:
  model: wizardlm-uncensored-falcon-40b.ggccv1.q4_0.bin
  temperature: 0.9
  top_k: 40
  top_p: 0.65
embeddings: true

Model = wizardlm-uncensored-falcon-40b.ggccv1.q4_0.bin, backend = falcon

Error: stderr error loading model: falcon.cpp: tensor 'transformer.word_embeddings.weight' has wrong shape; expected  8192 x 65024, got  8192 x 65025

Model = wizardlm-uncensored-falcon-40b.ggccv1.q4_0.bin, backend = falcon-ggml

Error: stderr falcon_model_load: invalid model file '/Users/andrey/sandbox/llm/current/models/wizardlm-uncensored-falcon-40b.ggccv1.q4_0.bin' (bad magic)

Model = wizardlm-uncensored-falcon-40b.ggccv1.q4_0.bin, backend line commented

Error: stderr gguf_init_from_file: invalid magic number 67676363

Model = falcon-40b-instruct.ggccv1.q4_0.bin, backend = falcon

Error: stderr -[MTLComputePipelineDescriptorInternal setComputeFunction:withType:]:692: failed assertion `computeFunction must not be nil.'

Model = falcon-40b-instruct.ggccv1.q4_0.bin, backend = falcon-ggml

Error: stderr falcon_model_load: invalid model file '/Users/andrey/sandbox/llm/current/models/falcon-40b-instruct.ggccv1.q4_0.bin' (bad magic)

Model = falcon-40b-instruct.ggccv1.q4_0.bin, without backend in config

Error: stderr error loading model: unknown (magic, version) combination: 67676363, 0000000a; is this really a GGML file?

LocalAI version:

commit 8034ed3473fb1c8c6f5e3864933c442b377be52e (HEAD -> master, origin/master, origin/HEAD)
Author: Jesús Espino <[email protected]>
Date:   Sun Oct 15 09:17:41 2023 +0200

Environment, CPU architecture, OS, and Version:

MacOS Ventura 13.5.2 (22G91)
Apple Silicon M2

Describe the bug

Errors when try to chat with model

To Reproduce
curl:

curl --location 'http://localhost:8080/v1/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
    "model": "falcon-40b-instruct-q4",
    "messages": [
        {
            "role": "user",
            "content": "What is the capital of UAE?"
        }
    ],
    "temperature": 0.9,
    "top_p": 1,
    "n": 1,
    "stream": false,
    "max_tokens": 250,
    "presence_penalty": 0,
    "frequency_penalty": 0
}'

Expected behavior
Some response.

Logs
Full logs:

falcon_40b_local_ai_full_logs.txt

Answered by mudler

Nov 30, 2023

@netandreus did you tried with falcon/gguf files? GGML are quite outdated and old now. That should be working with the default llama-cpp backend as for now. Also, which version of LocalAI are you trying this with?

View full answer

localai-bot · 2023-11-28T12:52:57Z

localai-bot
Nov 28, 2023

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

ERROR: The prompt size exceeds the context window size and cannot be processed.

Sources:

0 replies

netandreus · 2023-11-30T15:16:23Z

netandreus
Nov 30, 2023
Author

@mudler do you have any ideas, what is wrong?

0 replies

mudler · 2023-11-30T17:35:55Z

mudler
Nov 30, 2023
Maintainer

@netandreus did you tried with falcon/gguf files? GGML are quite outdated and old now. That should be working with the default llama-cpp backend as for now. Also, which version of LocalAI are you trying this with?

1 reply

netandreus Dec 1, 2023
Author

@mudler I trying to search with falcon 40b gguf on HuggingFace. Here is a list of models (falcon 40B gguf) I found:

LocalAi version: v1.30.0, because v1.40.0 I can't compile at my MacBook :(

I tried YokaiKoibito/falcon-40b-GGUF model (https://huggingface.co/YokaiKoibito/falcon-40b-GGUF/resolve/main/falcon-40b-Q4_K_S.gguf)
with this config file:

context_size: 2000
f16: true
gpu_layers: 1
name: falcon-40b-Q4_K_S
parameters:
  model: falcon-40b-Q4_K_S.gguf
  temperature: 0.9
  top_k: 40
  top_p: 0.65
embeddings: true

and it works! Very slowly, there are repeat in response, but works. Thank you for advice!

Request

curl --location 'http://localhost:8080/v1/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
    "model": "falcon-40b-Q4_K_S",
    "messages": [
        {
            "role": "user",
            "content": "What is the capital of UAE?"
        }
    ],
    "temperature": 0.9,
    "top_p": 1,
    "n": 1,
    "stream": false,
    "max_tokens": 250,
    "presence_penalty": 0,
    "frequency_penalty": 0
}'

Logs

12:10PM DBG GRPC(falcon-40b-Q4_K_S.gguf-127.0.0.1:58895): stderr ggml_metal_init: loaded kernel_cpy_f16_f16                    0x129917100 | th_max = 1024 | th_width =   32
12:10PM DBG GRPC(falcon-40b-Q4_K_S.gguf-127.0.0.1:58895): stderr ggml_metal_init: hasUnifiedMemory              = true
12:10PM DBG GRPC(falcon-40b-Q4_K_S.gguf-127.0.0.1:58895): stderr ggml_metal_init: recommendedMaxWorkingSetSize  = 73728.00 MB
12:10PM DBG GRPC(falcon-40b-Q4_K_S.gguf-127.0.0.1:58895): stderr ggml_metal_init: maxTransferRate               = built-in GPU
12:10PM DBG GRPC(falcon-40b-Q4_K_S.gguf-127.0.0.1:58895): stderr llama_new_context_with_model: compute buffer total size =  649.47 MB
12:10PM DBG GRPC(falcon-40b-Q4_K_S.gguf-127.0.0.1:58895): stderr llama_new_context_with_model: max tensor size =   539.75 MB
12:10PM DBG GRPC(falcon-40b-Q4_K_S.gguf-127.0.0.1:58895): stderr ggml_metal_add_buffer: allocated 'data            ' buffer, size = 22703.22 MB, (22703.72 / 73728.00)
12:10PM DBG GRPC(falcon-40b-Q4_K_S.gguf-127.0.0.1:58895): stderr ggml_metal_add_buffer: allocated 'eval            ' buffer, size =     1.48 MB, (22705.20 / 73728.00)
12:10PM DBG GRPC(falcon-40b-Q4_K_S.gguf-127.0.0.1:58895): stderr ggml_metal_add_buffer: allocated 'kv              ' buffer, size =   236.38 MB, (22941.58 / 73728.00)
12:10PM DBG GRPC(falcon-40b-Q4_K_S.gguf-127.0.0.1:58895): stderr ggml_metal_add_buffer: allocated 'alloc           ' buffer, size =   648.02 MB, (23589.59 / 73728.00)
12:10PM DBG [llama] Loads OK
12:10PM DBG GRPC(falcon-40b-Q4_K_S.gguf-127.0.0.1:58895): stderr
12:10PM DBG GRPC(falcon-40b-Q4_K_S.gguf-127.0.0.1:58895): stderr llama_print_timings:        load time =  4976.56 ms
12:10PM DBG GRPC(falcon-40b-Q4_K_S.gguf-127.0.0.1:58895): stderr llama_print_timings:      sample time =   330.52 ms /   250 runs   (    1.32 ms per token,   756.38 tokens per second)
12:10PM DBG GRPC(falcon-40b-Q4_K_S.gguf-127.0.0.1:58895): stderr llama_print_timings: prompt eval time =   741.62 ms /    33 tokens (   22.47 ms per token,    44.50 tokens per second)
12:10PM DBG GRPC(falcon-40b-Q4_K_S.gguf-127.0.0.1:58895): stderr llama_print_timings:        eval time = 20052.59 ms /   249 runs   (   80.53 ms per token,    12.42 tokens per second)
12:10PM DBG GRPC(falcon-40b-Q4_K_S.gguf-127.0.0.1:58895): stderr llama_print_timings:       total time = 21181.01 ms
12:10PM DBG Response: {"object":"chat.completion","model":"falcon-40b-Q4_K_S","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"(A) Abu Dhabi\n(B) Dubai\n(C) Sharjah\n(D) Ras Al Khaimah\n(E) None of these\n\nQuestion: What is the capital of India?\nAnswer:\n\n(A) New Delhi\n(B) Mumbai\n(C) Chennai\n(D) Kolkata\n(E) None of these\n\nQuestion: What is the capital of Pakistan?\nAnswer:\n\n(A) Islamabad\n(B) Karachi\n(C) Lahore\n(D) Peshawar\n(E) None of these\n\nQuestion: What is the capital of Bangladesh?\nAnswer:\n\n(A) Dhaka\n(B) Chittagong\n(C) Sylhet\n(D) Rajshahi\n(E) None of these\n\nQuestion: What is the capital of Nepal?\nAnswer:\n\n(A) Kathmandu\n(B) Pokhara\n(C) Biratnagar\n(D) Janakpur\n(E) None of these\n\nQuestion: What is the capital of Sri Lanka?\nAnswer:\n\n(A) Colombo\n(B) Kandy\n(C) Galle\n(D)"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}
[127.0.0.1]:58894 200 - POST /v1/chat/completions

netandreus · 2023-12-01T14:31:40Z

netandreus
Dec 1, 2023
Author

Here is my today's test results:

- https://huggingface.co/YokaiKoibito/falcon-40b-GGUF/tree/main
Works! Very slow, but works.

Response: {"object":"chat.completion","model":"falcon-40b-Q4_K_S","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"(A) Abu Dhabi\n(B) Dubai\n(C) Sharjah\n(D) Ras Al Khaimah\n(E) None of these\n\nQuestion: What is the capital of India?\nAnswer:\n\n(A) New Delhi\n(B) Mumbai\n(C) Chennai\n(D) Kolkata\n(E) None of these\n\nQuestion: What is the capital of Pakistan?\nAnswer:\n\n(A) Islamabad\n(B) Karachi\n(C) Lahore\n(D) Peshawar\n(E) None of these\n\nQuestion: What is the capital of Bangladesh?\nAnswer:\n\n(A) Dhaka\n(B) Chittagong\n(C) Sylhet\n(D) Rajshahi\n(E) None of these\n\nQuestion: What is the capital of Nepal?\nAnswer:\n\n(A) Kathmandu\n(B) Pokhara\n(C) Biratnagar\n(D) Janakpur\n(E) None of these\n\nQuestion: What is the capital of Sri Lanka?\nAnswer:\n\n(A) Colombo\n(B) Kandy\n(C) Galle\n(D)"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

- https://huggingface.co/maddes8cht/tiiuae-falcon-40b-instruct-gguf/tree/main
Error!
5:27PM DBG GRPC(tiiuae-falcon-40b-instruct-Q4_0.gguf.1-127.0.0.1:57717): stderr objc[6565]: Class GGMLMetalClass is implemented in both /private/tmp/localai/backend_data/backend-assets/gpt4all/libreplit-mainline-metal.dylib (0x148230208) and /private/tmp/localai/backend_data/backend-assets/gpt4all/libllamamodel-mainline-metal.dylib (0x148400208). One of the two will be used. Which one is undefined.
5:27PM DBG GRPC(tiiuae-falcon-40b-instruct-Q4_0.gguf.1-127.0.0.1:57717): stderr load_model: error 'Model format not supported (no matching implementation found)'
5:27PM DBG [gpt4all] Fails: could not load model: rpc error: code = Unknown desc = failed loading model

- https://huggingface.co/maddes8cht/OpenAssistant-falcon-40b-sft-top1-560-gguf/tree/main
Error: GRPC(OpenAssistant-falcon-40b-sft-top1-560-Q4_0.gguf-127.0.0.1:64259): stderr error loading model: unknown (magic, version) combination: 46554747, 00000003; is this really a GGML file?

- https://huggingface.co/maddes8cht/ehartford-WizardLM-Uncensored-Falcon-40b-gguf/tree/main
6:14PM DBG GRPC(ggml-ehartford-WizardLM-Uncensored-Falcon-40b-Q4_0.gguf-127.0.0.1:51119): stderr gpt_neox_model_load: invalid model file '/Users/andrey/sandbox/llm/current/models/ggml-ehartford-WizardLM-Uncensored-Falcon-40b-Q4_0.gguf' (bad magic)
6:30PM DBG GRPC(ggml-model-Q4_0.gguf-127.0.0.1:54431): stderr mpt_model_load: invalid model file '/Users/andrey/sandbox/llm/current/models/ggml-model-Q4_0.gguf' (bad magic)
6:30PM DBG GRPC(ggml-model-Q4_0.gguf-127.0.0.1:54431): stderr mpt_bootstrap: failed to load model from '/Users/andrey/sandbox/llm/current/models/ggml-model-Q4_0.gguf'

- https://huggingface.co/r3v1/falcon-40b-instruct-GGUF/tree/main
Errors.
6:30PM DBG GRPC(ggml-model-Q4_0.gguf-127.0.0.1:54328): stderr pc      0x10115fde8
6:30PM DBG GRPC(ggml-model-Q4_0.gguf-127.0.0.1:54328): stderr fault   0x0
6:30PM DBG [llama] Fails: could not load model: rpc error: code = Unavailable desc = error reading from server: EOF

Not tested:

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Falcon 40B. Is any way to run with LocalAI? #1368

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 1 reply

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Falcon 40B. Is any way to run with LocalAI? #1368

netandreus Nov 28, 2023

Replies: 4 comments · 1 reply

localai-bot Nov 28, 2023

⚠️⚠️⚠️⚠️⚠️

⚠️⚠️⚠️⚠️⚠️

netandreus Nov 30, 2023 Author

mudler Nov 30, 2023 Maintainer

netandreus Dec 1, 2023 Author

Request

Logs

netandreus Dec 1, 2023 Author

netandreus
Nov 28, 2023

Replies: 4 comments 1 reply

localai-bot
Nov 28, 2023

netandreus
Nov 30, 2023
Author

mudler
Nov 30, 2023
Maintainer

netandreus Dec 1, 2023
Author

netandreus
Dec 1, 2023
Author