llama : add Mixtral support

[slaren]

close #4381

Description

Add initial support for Mixture-of-Experts (MoE) LLM architectures.
Support for quantization and partial GPU offloading is available.

289443553-a3d5c7e3-db57-4ff8-90ce-59fcfc5b16d1.mp4

(vid) llama.cpp server running Q4_0 Mixtral-8x7B-32k on M2 Ultra

Running Mixtral-8x7B-32k

The following instructions work with the torrent data released on Dec 8.

# download torrent data into models/mixtral-8x7b-32k
# ...

# convert to F16
python3 convert.py ./models/mixtral-8x7b-32k/ \
         --outfile ./models/mixtral-8x7b-32k/ggml-model-f16.gguf \
         --outtype f16

# quantize to Q4_0
./quantize ./models/mixtral-8x7b-32k/ggml-model-f16.gguf \
           ./models/mixtral-8x7b-32k/ggml-model-q4_0.gguf \
           q4_0

# run Q4_0 inference
./main -m ./models/mixtral-8x7b-32k/ggml-model-q4_0.gguf \
       -p "I believe the meaning of life is" \
       -ngl 999 -s 1 -n 128 -t 8

Running the Instruct model

Download and convert:

# clone
git clone https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1 ./models/mixtral-instruct-8x7b

# convert to F16 GGUF
python3 convert.py ./models/mixtral-instruct-8x7b/ \
         --outfile ./models/mixtral-instruct-8x7b/ggml-model-f16.gguf \
         --outtype f16

# quantize to Q4_0
./quantize ./models/mixtral-instruct-8x7b/ggml-model-f16.gguf \
           ./models/mixtral-instruct-8x7b/ggml-model-q4_0.gguf \
           q4_0

Run like this for example:

./main \
  -m models/mixtral-instruct-8x7b/ggml-model-q4_0.gguf \
  -p "[INST] Prove that sqrt(2) is rational number. [/INST]" \
  --repeat_penalty 1 \
  --no-penalize-nl \
  --color --temp 0 -c 4096 -n -1 

llama_new_context_with_model: n_ctx      = 4096
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_new_context_with_model: KV self size  =  512.00 MiB, K (f16):  256.00 MiB, V (f16):  256.00 MiB
llama_build_graph: non-view tensors processed: 1124/1124
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M2 Ultra
ggml_metal_init: picking default device: Apple M2 Ultra
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: loading '/Users/ggerganov/development/github/llama.cpp/ggml-metal.metal'
ggml_metal_init: GPU name:   Apple M2 Ultra
ggml_metal_init: GPU family: MTLGPUFamilyApple8 (1008)
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 154618.82 MB
ggml_metal_init: maxTransferRate               = built-in GPU
llama_new_context_with_model: compute buffer total size = 319.35 MiB
llama_new_context_with_model: max tensor size =   102.54 MiB
ggml_metal_add_buffer: allocated 'data            ' buffer, size = 25216.62 MiB, (25218.25 / 147456.00)
ggml_metal_add_buffer: allocated 'kv              ' buffer, size =   512.03 MiB, (25730.28 / 147456.00)
ggml_metal_add_buffer: allocated 'alloc           ' buffer, size =   316.05 MiB, (26046.33 / 147456.00)

system_info: n_threads = 16 / 24 | AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | 
sampling: 
	repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
	top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.000
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order: 
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temp 
generate: n_ctx = 4096, n_batch = 512, n_predict = -1, n_keep = 0


 [INST] Prove that sqrt(2) is rational number. [/INST] I apologize, but the statement that sqrt(2) is a rational number is false. The number sqrt(2) is an irrational number, which means it cannot be expressed as a fraction a/b, where a and b are integers with no common factors other than 1, and b is not equal to zero.

To prove that sqrt(2) is irrational, we can use proof by contradiction. Here's a simple proof:

1. Assume, to the contrary, that sqrt(2) is rational. This means it can be expressed as a fraction a/b, where a and b are integers with no common factors other than 1, and b is not equal to zero.
2. If sqrt(2) = a/b, then squaring both sides gives 2 = a^2/b^2, or a^2 = 2 * b^2.
3. This equation implies that a^2 is even (since it's 2 times an integer). But then a must also be even (since only even numbers squared result in even numbers).
4. If a is even, then we can express a as 2k for some integer k. Substituting this into the equation gives (2k)^2 = 2 * b^2, or 4k^2 = 2 * b^2, or b^2 = 2 * k^2.
5. This equation implies that b^2 is even, and so b is also even.
6. But if both a and b are even, then they share a common factor of 2. This contradicts our initial assumption that a and b have no common factors other than 1.

Since our assumption that sqrt(2) is rational leads to a contradiction, we must conclude that sqrt(2) is irrational. [end of text]

llama_print_timings:        load time =   29022.56 ms
llama_print_timings:      sample time =       9.49 ms /   404 runs   (    0.02 ms per token, 42575.61 tokens per second)
llama_print_timings: prompt eval time =     179.90 ms /    20 tokens (    9.00 ms per token,   111.17 tokens per second)
llama_print_timings:        eval time =    7522.72 ms /   403 runs   (   18.67 ms per token,    53.57 tokens per second)
llama_print_timings:       total time =    7747.22 ms

Few notes:

• make sure to have enough context (-c 4096, can be even more, but note that default is only 512)
• disable the repeat penalty (--repeat_penalty 1), without this you can see typos, misspellings and early EOS
• disable the newline penalty (--no-penalize-nl), this might be important for code generation
• use -p "[INST] some instruction [/INST]", this should match the prompt template specified in the official repo

Implementation details

Supporting MoE in ggml requires the introduction of a new indirect matrix multiplication operator:

    // indirect matrix multiplication
    //  ggml_mul_mat_id(ctx, as, ids, id, b) ~= ggml_mul_mat(ctx, as[ids[id]], b)
    GGML_API struct ggml_tensor * ggml_mul_mat_id(
            struct ggml_context * ctx,
            struct ggml_tensor  * const as[],
            int                   n_as,
            struct ggml_tensor  * ids,
            int                   id,
            struct ggml_tensor  * b);

ggml_mul_mat_id allows selecting the source matrix dynamically during graph evaluation, based on the contents on the ids tensor, which can be the result of another operation. For batch evaluation, ids can contain multiple rows, and a different matrix is used to evaluate each row of the b matrix.

The current implementation is efficient for BS=1, but not so much for BS>1, since each token in the batch is evaluated separately. Improvements will follow up in the future.

Quantization support

The quantize tool can be used as usual to generate quantum versions of the model.

IMPORTANT NOTE
The currently implemented quantum mixtures are a first iteration and it is very likely to change in the future! Please, acknowledge that and be prepared to re-quantize or re-download the models in the near future!

Current quantum mixtures:

• The FFN tensors are quantized using the selected type
• F16 gating tensors (blk.{bid}.ffn_get_inp)
• Q8_0 KV tensors (blk.{bid}.attn_k.weight, blk.{bid}.attn_v.weight)

mixtral-q4_0-types.txt

GGUF changes

• add Keys.LLM.EXPERT_COUNT = "{arch}.expert_count"
• add Keys.LLM.EXPERT_USED_COUNT = "{arch}.expert_used_count"
• add 4 new tensor names:
  • FFN_GATE_INP
  • FFN_GATE_EXP
  • FFN_DOWN_EXP
  • FFN_UP_EXP

TODOs

• Document the GGUF changes in the spec: https://github.com/ggerganov/ggml/blob/master/docs/gguf.md (PRs welcome!)
• HF convert script update
• More robust conversions + better meta data

[jxy]

I'm not sure how @capdevc tested it, but I got short stopped answers from API, too.

$ curl --location "https://api.mistral.ai/v1/chat/completions" \
     --header 'Content-Type: application/json' \
     --header 'Accept: application/json' \
     --header "Authorization: Bearer $(GET_KEY_OBFUSCATED)" \
     --data '{
    "model": "mistral-small",
    "temperature": 0,
    "max_tokens": 1024,
    "messages": [
     {
        "role": "user",
        "content": "write a pong game using python make sure I can use the arrow keys to move."
      }
    ]
  }'                                      
{"id":"ID_OBFUSCATED","object":"chat.completion","created":1702657792,"model":"mistral-small","choices":[{"index":0,"message":{"role":"assistant","content":"Here is a simple implementation of Pong using the `curses` library in Python. This version allows the player to control the paddle using the arrow keys.\n```\nimport curses\n\n"},"finish_reason":"stop"}],"usage":{"prompt_tokens":26,"total_tokens":68,"completion_tokens":42}}

[ggerganov]

I'm not sure how @capdevc tested it

HF chat either has a system prompt or does not use temp = 0 so the test is invalid. Your results confirm that it's not a llama.cpp problem, but a model behaviour.

[capdevc]

@jxy @ggerganov

Just confirmed, that I called the API with a non-zero temp due to a screwup with some defaults handling on my part. Replicating with curl directly, I get the same result you two did. Apologies for the noise there.

Also, calling the Q8 via llama.cpp directly gives the same truncated result:

./main -m ../../../Models/mixtral-8x7b-instruct-v0.1.Q8_0.gguf --color --temp 0 --repeat_penalty 1 -c 32768 -n -1 -p "[INST] write a pong game using python make sure I can use the arrow keys to move. [/INST]"

gives

 [INST] write a pong game using python make sure I can use the arrow keys to move. [/INST] Here is a simple implementation of Pong using the `curses` library in Python. This version allows the player to control the paddle using the arrow keys.
```
import curses

 [end of text]

[Rotatingxenomorph]

@jxy @ggerganov

Just confirmed, that I called the API with a non-zero temp due to a screwup with some defaults handling on my part. Replicating with curl directly, I get the same result you two did. Apologies for the noise there.

Also, calling the Q8 via llama.cpp directly gives the same truncated result:

./main -m ../../../Models/mixtral-8x7b-instruct-v0.1.Q8_0.gguf --color --temp 0 --repeat_penalty 1 -c 32768 -n -1 -p "[INST] write a pong game using python make sure I can use the arrow keys to move. [/INST]"

gives

 [INST] write a pong game using python make sure I can use the arrow keys to move. [/INST] Here is a simple implementation of Pong using the `curses` library in Python. This version allows the player to control the paddle using the arrow keys.

import curses

[end of text]

That's weird. The Q8 works for me, and the sha256 is correct.

[Rotatingxenomorph]

I think my issue was with the latest build of avx2 for windows llamacpp.

It acts completely differently (way worse) than the cublas version I have main: build = 1629 (799a1cb)
main: built with MSVC 19.37.32826.1 for x64

Edit the latest cublas works okay.

For this prompt:

mixtral-8x7b-instruct-v0.1.Q8_0.gguf --top-p 1 --color -t 5 --temp 3 --repeat_penalty 1.2 -c 4096 -n -1 --min-p 0.050 -s 1702748009 -p "[INST] --------------------------------------------------

You are an expert in analogies.

Marathon is to race as hibernation is to

winter

bear

dream

sleep

Think this through logically step by step. [/INST]"

[brozkrut]

M2 Max Studio, 8+4 CPU, 38 GPU, 96 GB - Mixtral 8x 7B Instruct

I ran some benchmarks for Mixtral Instruct - here are the results:

Model (Build: 9fb13f9)	Size	PPL (final estimate)	pp 512	tg 128
llama 7B mostly Q2_K	14.57 GiB	7.0648 +/- 0.04209	75.22 ± 0.01	39.89 ± 0.03
llama 7B mostly Q3_K - Medium	18.96 GiB	4.6644 +/- 0.02530	41.19 ± 0.00	28.35 ± 0.01
llama 7B mostly Q4_K - Medium	24.62 GiB	4.5111 +/- 0.02423	67.72 ± 0.01	34.72 ± 0.03
llama 7B mostly Q4_0	24.62 GiB	4.5087 +/- 0.02428	89.41 ± 0.01	36.66 ± 0.05
llama 7B mostly Q5_K - Medium	30.02 GiB	4.4396 +/- 0.02379	42.19 ± 0.00	26.06 ± 0.02
llama 7B mostly Q5_0	30.02 GiB	4.4345 +/- 0.02373	44.15 ± 0.00	28.43 ± 0.02
llama 7B mostly Q6_K	35.74 GiB	4.424 +/- 0.02368	40.27 ± 0.01	25.44 ± 0.01
llama 7B mostly Q8_0	46.22 GiB	4.4088 +/- 0.02353	67.48 ± 0.01	22.87 ± 0.01

Prompt Processing:

Text Generation:

---

% ./main --version
version: 1631 (9fb13f9)
built with Apple clang version 15.0.0 (clang-1500.1.0.2.5) for arm64-apple-darwin23.1.0

# https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF/tree/main
# https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-raw-v1.zip?ref=salesforce-research
% shasum -a 256 mixtral-8x7b-instruct-v0.1.Q* wikitext-2-raw/wiki.test.raw
d54b4f4ec06dbae558d25b2d1542417cdf9547907342db85eecd05b6e96e88f8  mixtral-8x7b-instruct-v0.1.Q2_K.gguf
bd2e1499e68195f1a6ff151e6fa5c6632acc150b80cca4a3772cbb7ca59d44cd  mixtral-8x7b-instruct-v0.1.Q3_K_M.gguf
0c57465507f21bed4364fca37efd310bee92e25a4ce4f5678ef9b44e95830e4e  mixtral-8x7b-instruct-v0.1.Q4_0.gguf
9193684683657e90707087bd1ed19fd0b277ab66358d19edeadc26d6fdec4f53  mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf
6c4dd1082dfa8d89f901039c7095f8b4343dca7a6782c82a72decb6a44475803  mixtral-8x7b-instruct-v0.1.Q5_0.gguf
af12961e014037ee8c5c9f3bf7cf9fd99cadc9dabd50f528a4248c4a8ee8fe77  mixtral-8x7b-instruct-v0.1.Q5_K_M.gguf
56638f9853b8fff80ac1fd4a91434a1c15c21d4c910811c5458df9ef092615fd  mixtral-8x7b-instruct-v0.1.Q6_K.gguf
cdca4a8c09dfd722702f781d479695cda0d45e1bd1cd602ba1b6085ad921fc5f  mixtral-8x7b-instruct-v0.1.Q8_0.gguf
173c87a53759e0201f33e0ccf978e510c2042d7f2cb78229d9a50d79b9e7dd08  wikitext-2-raw/wiki.test.raw

% ./perplexity -m models/mixtral-8x7b-32k/mixtral-8x7b-instruct-v0.1.Q4_0.gguf -f wikitext-2-raw/wiki.test.raw

% ./llama-bench \
  -m ./models/mixtral-8x7b-instruct-v0.1.Q2_K.gguf \
  -m ./models/mixtral-8x7b-instruct-v0.1.Q3_K_M.gguf \
  -m ./models/mixtral-8x7b-instruct-v0.1.Q4_0.gguf \
  -m ./models/mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf \
  -m ./models/mixtral-8x7b-instruct-v0.1.Q5_0.gguf \
  -m ./models/mixtral-8x7b-instruct-v0.1.Q5_K_M.gguf \
  -m ./models/mixtral-8x7b-instruct-v0.1.Q6_K.gguf \
  -m ./models/mixtral-8x7b-instruct-v0.1.Q8_0.gguf \
  -p 512 -n 128 -ngl 99 2> /dev/null

[shuhongwu]

just tested Mixtral 8 x 7B on M2 Ultra 192G with Q4_K_M, thanks for you hard work.
https://twitter.com/AlexWuKing/status/1736247587404210600
And I also tested Q8 on M2 Ultra, I can confidently say that Mixtral 8 x 7B with Q8 can defeat ChatGPT3.5

[moshemalawach]

Any BLAS available to speed up context processing with this model (cublas/clblast/openblas)? Haven't been able to get it to work.

[toncho11]

I have only 16 GB of memory.
Would it work on CPU? My GPU is only 4 GB.

[teleprint-me]

@toncho11 Yes, you'll be fine. Full 32k context might not fit though, something to keep in mind. Play around with the values and see what works for you.

[Limezero]

Could anyone clarify what the current state of Mixtral support in llama.cpp is?

I've seen a ton of conflicting information out there saying that the models are broken, k-quants have unusually high perplexity so we shouldn't use them, BLAS and prompt processing acceleration doesn't work, GPU support doesn't work, Mixtral generates worse results than a 7B, etc., so I'm hoping someone knowledgeable and up-to-date on this stuff could chip in.

[eugenepyvovarov]

Could anyone clarify what the current state of Mixtral support in llama.cpp is?

I've seen a ton of conflicting information out there saying that the models are broken, k-quants have unusually high perplexity so we shouldn't use them, BLAS and prompt processing acceleration doesn't work, GPU support doesn't work, Mixtral generates worse results than a 7B, etc., so I'm hoping someone knowledgeable and up-to-date on this stuff could chip in.

works great on m1 ultra 128g, quantified

[ggerganov]

I've seen a ton of conflicting information out there saying that ...

Whatever information you see out there you can safely assume it is wrong unless some specific example is provided. There are many little details involved in using LLMs correctly and the chance of getting something wrong is very high. My advice is do your own tests and make your own conclusions

[Limezero]

Whatever information you see out there you can safely assume it is wrong unless some specific example is provided. There are many little details involved in using LLMs correctly and the chance of getting something wrong is very high. My advice is do your own tests and make your own conclusions

Are there no obvious "oh yeah we know X is broken/not implemented/performs worse than it should, someone is working on a PR" caveats with Mixtral support currently, then?

[ggerganov]

Other than what is already written in OP - no

[shuhongwu]

Could anyone clarify what the current state of Mixtral support in llama.cpp is?
I've seen a ton of conflicting information out there saying that the models are broken, k-quants have unusually high perplexity so we shouldn't use them, BLAS and prompt processing acceleration doesn't work, GPU support doesn't work, Mixtral generates worse results than a 7B, etc., so I'm hoping someone knowledgeable and up-to-date on this stuff could chip in.

works great on m1 ultra 128g, quantified

Yes, I tried on M2 Ultra with 192GB unified memory and it almost have 150GB GPU memory, tested with Mixtral 8x7B Q8, it give me the impression that it can be comparable to the ChatGPT 3.5.

[clemens98]

Could anyone clarify what the current state of Mixtral support in llama.cpp is?

I've seen a ton of conflicting information out there saying that the models are broken, k-quants have unusually high perplexity so we shouldn't use them, BLAS and prompt processing acceleration doesn't work, GPU support doesn't work, Mixtral generates worse results than a 7B, etc., so I'm hoping someone knowledgeable and up-to-date on this stuff could chip in.

For me it Matches perfectly
K quants perform extremely Poor(q5k_m was worse than Mistral 7b)
4.0 quantization works very Good has long gpu offloading is set to Zero
Clblast doesn't work
Don't know if ROCm works

[pudepiedj]

Could anyone clarify what the current state of Mixtral support in llama.cpp is?
I've seen a ton of conflicting information out there saying that the models are broken, k-quants have unusually high perplexity so we shouldn't use them, BLAS and prompt processing acceleration doesn't work, GPU support doesn't work, Mixtral generates worse results than a 7B, etc., so I'm hoping someone knowledgeable and up-to-date on this stuff could chip in.

For me it Matches perfectly K quants perform extremely Poor(q5k_m was worse than Mistral 7b) 4.0 quantization works very Good has long gpu offloading is set to Zero Clblast doesn't work Don't know if ROCm works

I have been running mixtral-8x7b-instruct-v0.1.Q_K_M.gguf on a 32GB M2 MAX by increasing the gpu allocation using the sudo sysctl iogpu.wired_limit_mb=27500 trick and -ngl 99 with -c 4096 and -i -ins and the performance has been very good. Slightly less good using -p prompting in a python loop. Speed is around 24 t/s and quality of responses is high. Here's what I use:

./bin/main -m ../models/Mixtral-8x7b/mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf -c 4096 -ngl 99 -n -1 -s 1 -i -ins -ctk q8_0 --override-kv llama.expert_used_count=int:3

[clemens98]

I have been running mixtral-8x7b-instruct-v0.1.Q_K_M.gguf on a 32GB M2 MAX by increasing the gpu allocation using the sudo sysctl iogpu.wired_limit_mb=27500 trick and -ngl 99 with -c 4096 and -i -ins and the performance has been very good. Slightly less good using -p prompting in a python loop. Speed is around 24 t/s and quality of responses is high. Here's what I use:

./bin/main -m ../models/Mixtral-8x7b/mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf -c 4096 -ngl 99 -n -1 -s 1 -i -ins -ctk q8_0 --override-kv llama.expert_used_count=int:3

Is that clblast or is that the apple equivalent ?
I am using a rx7900xt and a Ryzen 5600

The iogpu Wired limit seams to be an Apple thing

[pudepiedj]

I have been running mixtral-8x7b-instruct-v0.1.Q_K_M.gguf on a 32GB M2 MAX by increasing the gpu allocation using the sudo sysctl iogpu.wired_limit_mb=27500 trick and -ngl 99 with -c 4096 and -i -ins and the performance has been very good. Slightly less good using -p prompting in a python loop. Speed is around 24 t/s and quality of responses is high. Here's what I use:

./bin/main -m ../models/Mixtral-8x7b/mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf -c 4096 -ngl 99 -n -1 -s 1 -i -ins -ctk q8_0 --override-kv llama.expert_used_count=int:3

Is that clblast or is that the apple equivalent ? I am using a rx7900xt and a Ryzen 5600

The iogpu Wired limit seams to be an Apple thing

Yes, this is on Apple M2 MAX silicon using the ggml metal implementation.

[clemens98]

Someone on Reddit sayd Mixtral acceleration isn't supported on AMD GPUs . Is that True?

[pudepiedj]

Someone on Reddit sayd Mixtral acceleration isn't supported on AMD GPUs . Is that True?

Short answer, at least on my hardware: yes. But even if it were supported, the hardware can't deal with it; at least, my 16GB Intel i9 8-core CPU and 4GB AMD Radeon Pro 5500M GPU can't.

Lots of mat_mul kernels are not supported (see below), but even if they were there are bigger problems.
Here's an ls -l ../models/Mixtral-8x7b report:

-rw-r--r--@ 1 edsil  staff  26441533376 13 Dec 08:57 mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf
-rw-r--r--@ 1 edsil  staff  15644034176 11 Dec 16:48 mixtral-8x7b-v0.1.Q2_K.gguf
-rw-r--r--@ 1 edsil  staff  20363355584 12 Dec 10:50 mixtral-8x7b-v0.1.Q3_K_M.gguf

So mixtral-8x7b-v0.1.Q2_K.gguf is about 15GB and if I try to load it on my 2019 16/4 Intel/AMD Radeon Pro 5500M` using

./bin/main -m ../models/Mixtral-8x7b/mixtral-8x7b-v0.1.Q2_K.gguf -ngl 7 -c 2048 -n 2048 -p "What is the meaning of life?"

it tries to use (14395.95 / 4080.00) of VRAM! :)

llama_new_context_with_model: n_ctx      = 2048
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Intel(R) UHD Graphics 630
ggml_metal_init: found device: AMD Radeon Pro 5500M
ggml_metal_init: picking default device: AMD Radeon Pro 5500M
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: loading '/Users/edsil/llama.cpp/build/bin/ggml-metal.metal'
ggml_metal_init: GPU name:   AMD Radeon Pro 5500M
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction support   = true
ggml_metal_init: simdgroup matrix mul. support = false
ggml_metal_init: hasUnifiedMemory              = false
ggml_metal_init: recommendedMaxWorkingSetSize  =  4278.19 MB
ggml_metal_init: maxTransferRate               = built-in GPU
ggml_metal_init: skipping kernel_mul_mm_f32_f32            (not supported)
ggml_metal_init: skipping kernel_mul_mm_f16_f32            (not supported)
ggml_metal_init: skipping kernel_mul_mm_q4_0_f32           (not supported)
ggml_metal_init: skipping kernel_mul_mm_q4_1_f32           (not supported)
ggml_metal_init: skipping kernel_mul_mm_q5_0_f32           (not supported)
ggml_metal_init: skipping kernel_mul_mm_q5_1_f32           (not supported)
ggml_metal_init: skipping kernel_mul_mm_q8_0_f32           (not supported)
ggml_metal_init: skipping kernel_mul_mm_q2_K_f32           (not supported)
ggml_metal_init: skipping kernel_mul_mm_q3_K_f32           (not supported)
ggml_metal_init: skipping kernel_mul_mm_q4_K_f32           (not supported)
ggml_metal_init: skipping kernel_mul_mm_q5_K_f32           (not supported)
ggml_metal_init: skipping kernel_mul_mm_q6_K_f32           (not supported)
ggml_metal_init: skipping kernel_mul_mm_iq2_xxs_f32        (not supported)
ggml_metal_init: skipping kernel_mul_mm_iq2_xs_f32         (not supported)
ggml_metal_init: skipping kernel_mul_mm_id_f32_f32         (not supported)
ggml_metal_init: skipping kernel_mul_mm_id_f16_f32         (not supported)
ggml_metal_init: skipping kernel_mul_mm_id_q4_0_f32        (not supported)
ggml_metal_init: skipping kernel_mul_mm_id_q4_1_f32        (not supported)
ggml_metal_init: skipping kernel_mul_mm_id_q5_0_f32        (not supported)
ggml_metal_init: skipping kernel_mul_mm_id_q5_1_f32        (not supported)
ggml_metal_init: skipping kernel_mul_mm_id_q8_0_f32        (not supported)
ggml_metal_init: skipping kernel_mul_mm_id_q2_K_f32        (not supported)
ggml_metal_init: skipping kernel_mul_mm_id_q3_K_f32        (not supported)
ggml_metal_init: skipping kernel_mul_mm_id_q4_K_f32        (not supported)
ggml_metal_init: skipping kernel_mul_mm_id_q5_K_f32        (not supported)
ggml_metal_init: skipping kernel_mul_mm_id_q6_K_f32        (not supported)
ggml_metal_init: skipping kernel_mul_mm_id_iq2_xxs_f32     (not supported)
ggml_metal_init: skipping kernel_mul_mm_id_iq2_xs_f32      (not supported)
llama_kv_cache_init:        CPU KV buffer size =   200.00 MiB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =    56.00 MiB, (14395.95 /  4080.00)ggml_backend_metal_buffer_type_alloc_buffer: warning: current allocated size is greater than the recommended max working set size
llama_kv_cache_init:      Metal KV buffer size =    56.00 MiB
llama_new_context_with_model: KV self size  =  256.00 MiB, K (f16):  128.00 MiB, V (f16):  128.00 MiB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     0.00 MiB, (14395.95 /  4080.00)ggml_backend_metal_buffer_type_alloc_buffer: warning: current allocated size is greater than the recommended max working set size
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =   192.01 MiB, (14587.96 /  4080.00)ggml_backend_metal_buffer_type_alloc_buffer: warning: current allocated size is greater than the recommended max working set size
llama_new_context_with_model: graph splits (measure): 5
llama_new_context_with_model:      Metal compute buffer size =   192.01 MiB
llama_new_context_with_model:        CPU compute buffer size =   184.04 MiB

If I change the model to a Q8_0.gguf version of llama-2-7b it runs - albeit slowly.

[clemens98]

Someone on Reddit sayd Mixtral acceleration isn't supported on AMD GPUs . Is that True?

Short answer, at least on my hardware: yes. But even if it were supported, the hardware can't deal with it; at least, my 16GB Intel i9 8-core CPU and 4GB AMD Radeon Pro 5500M GPU can't.

Lots of mat_mul kernels are not supported (see below), but even if they were there are bigger problems. Here's an ls -l ../models/Mixtral-8x7b report:

-rw-r--r--@ 1 edsil  staff  26441533376 13 Dec 08:57 mixtral-8x7b-instruct-v0.1.Q4_K_M.gguf
-rw-r--r--@ 1 edsil  staff  15644034176 11 Dec 16:48 mixtral-8x7b-v0.1.Q2_K.gguf
-rw-r--r--@ 1 edsil  staff  20363355584 12 Dec 10:50 mixtral-8x7b-v0.1.Q3_K_M.gguf

So mixtral-8x7b-v0.1.Q2_K.gguf is about 15GB and if I try to load it on my 2019 16/4 Intel/AMD Radeon Pro 5500M` using

./bin/main -m ../models/Mixtral-8x7b/mixtral-8x7b-v0.1.Q2_K.gguf -ngl 7 -c 2048 -n 2048 -p "What is the meaning of life?"

it tries to use (14395.95 / 4080.00) of VRAM! :)

Very Strange my 4.0 Mixtral uses 17gb with NGL 13-15