Apple: LocalAI vs. LM Studio - why is LM Studio so much faster?

[rozek]

Hello!

First of all: thank you very much for LocalAI!

I am currently experimenting with LocalAI and LM Studio on an Macbook Air with M2 and 24GB RAM - both controlled using FlowiseAI

Surprisingly, whenever I use LM Studio with the same settings (particularly, the same model, namely llama-2-13b.ggmlv3.q4_0), LM Studio turns out to

• be much faster,
• produce better output and
• be able to stop itself (before running into the token generation limit).

The prompts sent to both services (I'm using LM Studio as a service) should be identical, thus, there should only be two potential differences left:

• the precise parameters sent to llama.cpp (if that is used) and
• potential support for the Mx GPUs

I built LocalAI for Macs with Metal support (make BUILD_TYPE=metal build) but could not recognize a substantial acceleration - on the other side, I do not really understand the two lines

# Set `gpu_layers: 1` to your YAML model config file and `f16: true`
# Note: only models quantized with q4_0 are supported!

I used a base.yaml description as a start, but where should those two settings be added?

[XYWENJIE]

Yes, I have been using the same model with the same parameters recently, and the output quality in LM Studio is much higher than that in LoadAI.

[mudler]

really depends how you are running local-ai. For instance, to benefit from the Apple/Mac optimizations you should run it as a binary instead of using docker. Otherwise rosetta would kick in and slow down things. Another aspect would probably be metal support.

Given people reporting it #1064 to be broken that's the real slowdown you are seeing there - but unfortunately I don't own an apple so It's hard for me to test and check out what's wrong there. If someone with a mac and a bit of patience would like to help there we could get that fixed.