illegal instrution

[adaaaaaa]

system:android 13
python:3.11
model:vicuna-7b-v1.3.ggmlv3.q4_1.bin

`~/.../models/7B $ ln -s ~/storage/downloads/python/vicuna-7b-v1.3.ggmlv3.q4_1.bin ggml-model.bin

~/.../models/7B $ ls
ggml-model.bin ggml-model.bin.old

~/.../models/7B $ cd ../..

~/llama.cpp $ ./main
main: build = 776 (55dbb91)
main: seed = 1688404061
llama.cpp: loading model from models/7B/ggml-model.bin
llama_model_load_internal: format = ggjt v3 (latest)
llama_model_load_internal: n_vocab = 32000 llama_model_load_internal: n_ctx = 512 llama_model_load_internal: n_embd = 4096
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 32
llama_model_load_internal: n_layer = 32 llama_model_load_internal: n_rot = 128 llama_model_load_internal: ftype = 3 (mostly Q4_1)
llama_model_load_internal: n_ff = 11008 llama_model_load_internal: model size = 7B llama_model_load_internal: ggml ctx size = 0.08 MB
Illegal instruction`

what' the next?

[mqy]

If you can use cmake, try the following, may print more diagnose info.

mkdir -p build
rm -rf build/*
cd build
cmake .. -DLLAMA_SANITIZE_ADDRESS=ON && cmake --build . --config Debug
cd ..
./build/bin/main ...

[adaaaaaa]

@mqy
`~/llama.cpp/build $ cmake .. -DLLAMA_SANITIZE_ADDRESS=ON && cmake -- The C compiler identification is Clang 16.0.6
-- The CXX compiler identification is Clang 16.0.6
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /data/data/com.termux/files/usr/bin/cc - skipped -- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /data/data/com.termux/files/usr/bin/c++ - skipped -- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /data/data/com.termux/files/usr/bin/git (found version "2.41.0")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- CMAKE_SYSTEM_PROCESSOR: aarch64 -- ARM detected
-- Configuring done (9.0s)
-- Generating done (1.6s)
-- Build files have been written to: /home/llama.cpp/build
Usage
cmake [options]
cmake [options]
cmake [options] -S -B
Specify a source directory to (re-)generate a build system for it in the
current working directory. Specify an existing build directory to
re-generate its build system. Run 'cmake --help' for more information.

~/llama.cpp/build $ ls CMakeCache.txt bin
CMakeFiles cmake_install.cmake
CTestTestfile.cmake compile_commands.json DartConfiguration.tcl examples Makefile pocs
Testing tests
~/llama.cpp/build $ cd ..
~/llama.cpp $ ./build/bin/main ...
bash: ./build/bin/main: No such file or directory
~/llama.cpp $ ./build/bin/main
bash: ./build/bin/main: No such file or directory
~/llama.cpp $ cd build/bin/
~/.../build/bin $ ls
~/.../build/bin $`

[mqy]

you didn't run cmake --build . --config Debug at all

[ghost]

system:android 13 python:3.11 model:vicuna-7b-v1.3.ggmlv3.q4_1.bin

`~/.../models/7B $ ln -s ~/storage/downloads/python/vicuna-7b-v1.3.ggmlv3.q4_1.bin ggml-model.bin

~/.../models/7B $ ls ggml-model.bin ggml-model.bin.old

~/.../models/7B $ cd ../..

what' the next?

"the next" is building llama.cpp, then moving the GGML to the correct folder. Do not load a GGML from the downloads folder.

Here are some steps to follow:

1. Clone the repo:

$HOME

git clone https://github.com/ggerganov/llama.cpp

2. Move your model to the proper directory, for example:

cd storage/downloads
mv 7b-ggml-q4_0.bin ~/llama.cpp/models

3. e. Build llama.cpp:

cd llama.cpp

make

Here's an example of using llama.cpp:

./main -m ~/llama.cpp/models/3b-ggml-q8_0.bin --color -c 2048 --keep -1 -t 3 -b 10 -i -ins

*modify -t # parameter to the # of physical cores in your device.

If you have an issue, then let me know precisely what step you made it to, and the error.

[adaaaaaa]

@mqy @JackJollimore
here's the output ,thanks!

[adaaaaaa]

@JackJollimore

~/llama.cpp $ make clean
I llama.cpp build info:                                 
I UNAME_S:  Linux
I UNAME_P:  unknown                                     
I UNAME_M:  aarch64                                     
I CFLAGS:   -I.              -O3 -std=c11   -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -DGGML_USE_K_QUANTS -mcpu=native
I CXXFLAGS: -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_K_QUANTS -mcpu=native                                                   I LDFLAGS:                                              
I CC:       clang version 16.0.6
I CXX:      clang version 16.0.6                                                                                rm -vf *.o *.so main quantize quantize-stats perplexity embedding benchmark-matmult save-load-state server vdot train-text-from-scratch embd-input-test build-info.h    
removed 'common.o'
removed 'ggml.o'                                        
removed 'k_quants.o'                                    
removed 'llama.o'                                       
removed 'libembdinput.so'                               
removed 'main'                                          
removed 'quantize'                                      
removed 'quantize-stats'                                
removed 'perplexity'                                    
removed 'embedding'                                     
removed 'vdot'                                          
removed 'train-text-from-scratch'
removed 'embd-input-test'                               
removed 'build-info.h'~/llama.cpp $ cp ~/storage/downloads/python/vicuna-7b-v1.3.ggmlv3.q4_1.bin models/
~/llama.cpp $ make                                      
I llama.cpp build info:                                 
I UNAME_S:  Linux                                       
I UNAME_P:  unknown
I UNAME_M:  aarch64                                     
I CFLAGS:   -I.              -O3 -std=c11   -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -DGGML_USE_K_QUANTS -mcpu=native                       
I CXXFLAGS: -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_K_QUANTS -mcpu=native                                                   I LDFLAGS:
I CC:       clang version 16.0.6                        
I CXX:      clang version 16.0.6                                                                                cc  -I.              -O3 -std=c11   -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -DGGML_USE_K_QUANTS -mcpu=native   -c ggml.c -o ggml.o         
aarch64-linux-android-clang++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_K_QUANTS -mcpu=native -c llama.cpp -o llama.o         
aarch64-linux-android-clang++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_K_QUANTS -mcpu=native -c examples/common.cpp -o common.o
cc -I.              -O3 -std=c11   -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -DGGML_USE_K_QUANTS -mcpu=native   -c -o k_quants.o k_quants.c
aarch64-linux-android-clang++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_K_QUANTS -mcpu=native examples/main/main.cpp ggml.o llama.o common.o k_quants.o -o main                                                                               ====  Run ./main -h for help.  ====                     
aarch64-linux-android-clang++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_K_QUANTS -mcpu=native examples/quantize/quantize.cpp ggml.o llama.o k_quants.o -o quantize                    
aarch64-linux-android-clang++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_K_QUANTS -mcpu=native examples/quantize-stats/quantize-stats.cpp ggml.o llama.o k_quants.o -o quantize-stats  aarch64-linux-android-clang++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_K_QUANTS -mcpu=native examples/perplexity/perplexity.cpp ggml.o llama.o common.o k_quants.o -o perplexity     
aarch64-linux-android-clang++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_K_QUANTS -mcpu=native examples/embedding/embedding.cpp ggml.o llama.o common.o k_quants.o -o embedding        
aarch64-linux-android-clang++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_K_QUANTS -mcpu=native pocs/vdot/vdot.cpp ggml.o k_quants.o -o vdot                                            aarch64-linux-android-clang++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_K_QUANTS -mcpu=native examples/train-text-from-scratch/train-text-from-scratch.cpp ggml.o llama.o k_quants.o -o train-text-from-scratch                               
aarch64-linux-android-clang++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_K_QUANTS -mcpu=native examples/simple/simple.cpp ggml.o llama.o common.o k_quants.o -o simple                 
aarch64-linux-android-clang++ --shared -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_K_QUANTS -mcpu=native examples/embd-input/embd-input-lib.cpp ggml.o llama.o common.o k_quants.o -o libembdinput.so                                           
aarch64-linux-android-clang++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -DGGML_USE_K_QUANTS -mcpu=native examples/embd-input/embd-input-test.cpp ggml.o llama.o common.o k_quants.o -o embd-input-test  -L. -lembdinput
~/llama.cpp $ ./main -m ~/llama.cpp/models/vicuna-7b-v1.3.ggmlv3.q4_1.bin --color -c 2048 --keep -1 -t 3 -b 10 -i -ins                                                  
Illegal instruction

[Green-Sky]

@adaaaaaa can you run uname -a and tell us the output?
edit: and lscpu

[ghost]

~/llama.cpp $ ./main -m ~/llama.cpp/models/vicuna-7b-v1.3.ggmlv3.q4_1.bin --color -c 2048 --keep -1 -t 3 -b 10 -i -ins
Illegal instruction

@adaaaaaa

Thanks for your response. Please let us know -uname -a and lscpu.

I have to ask just in case: You're using Termux from Fdroid, yah?

Edit:

Output shows that you copied the GGML to models, which the wrong directory. The correct path is ~/llama.cpp/models.

[adaaaaaa]

@JackJollimore yes,termux
@Green-Sky
/dev $ uname -a && lscpu
Linux localhost 5.10.101-android12-9-00005-ga829d48e78bd-ab9206161 #1 SMP PREEMPT Fri Oct 21 21:49:09 UTC 2022 aarch64 Android
lscpu: failed to determine number of CPUs: /sys/devices/system/cpu/possible: No such file or directory

cpu is snapdrogan8+gen1

[adaaaaaa]

@JackJollimore
nope,i had copied model file to llama.cpp/models
`~/llama.cpp/models $ ls

7B ggml-vocab.bin vicuna-7b-v1.3.ggmlv3.q4_1.bin
`

[ghost]

@JackJollimore yes,termux

Sorry, I'm not certain based on your response.

Termux from the Google Playstore is incompatible.

Termux from FDroid is required.

[mqy]

@mqy @JackJollimore here's the output ,thanks!

I did not sawsee Illegal in the pasted content though. Instead, the program works fine --
the./bin/main random output some codes, right?

[adaaaaaa]

@mqy

random output some codes

yes,i run it the second time ,it output different codes now,why?

[adaaaaaa]

Termux from FDroid is required.

it's download from droid-ify...not google.

[ghost]

@mqy @JackJollimore here's the output ,thanks!

According to the output, llama.cpp works as expected from build/bin. There's no illegal instruction.

Edit: please ensure Output is readable going forward.

lscpu: failed to determine number of CPUs: /sys/devices/system/cpu/possible: No such file or directory`

is Termux is fully setup?

termux-setup-storage

termux-change-repo

apt update && apt upgrade

[mqy]

Since main from cmake works, I'm guessing the Illegal instruction is caused by an outdated main binary.

Here is the stand output pattern:

llama_model_load_internal: ggml ctx size =    0.06 MB
llama_model_load_internal: mem required  = 2862.72 MB (+  682.00 MB per state)

@adaaaaaa 's main crashed right after the first line.

@adaaaaaa would you please build mian with make again and run it?

BTW, you'd better tell us the latest commit (git log -1), it's good to follow standard bug report process, isn't?

[adaaaaaa]

@mqy
build again and get the same result,by the way,i have rm -rf ./llama.cpp/build
~/llama.cpp $ ./main
main: build = 786 (b472f3f)
main: seed = 1688564534
llama.cpp: loading model from models/7B/ggml-model.bin
llama_model_load_internal: format = ggjt v3 (latest)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 512
llama_model_load_internal: n_embd = 4096
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 32
llama_model_load_internal: n_layer = 32
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 3 (mostly Q4_1)
llama_model_load_internal: n_ff = 11008
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size = 0.08 MB
Illegal instruction

~/llama.cpp $ git log -1
commit b472f3f (HEAD -> master, origin/master, origin/HEAD)
Author: Georgi Gerganov [email protected]
Date: Tue Jul 4 22:25:22 2023 +0300
readme : add link web chat PR

[adaaaaaa]

@JackJollimore

is Termux is fully setup?

yes,and had updated yesterday

[adaaaaaa]

@Green-Sky @JackJollimore
~/llama.cpp $ lscpu
Architecture: aarch64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Vendor ID: ARM
Model name: Cortex-A510
Model: 3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
Stepping: r0p3
CPU(s) scaling MHz: 45%
CPU max MHz: 1804.8000
CPU min MHz: 300.0000
BogoMIPS: 38.40
Flags: fp asimd evtstrm aes pmull sha1
sha2 crc32 atomics fphp asimdhp
cpuid asimdrdm jscvt fcma lrcpc
dcpop sha3 sm3 sm4 asimddp sha51
2 asimdfhm dit uscat ilrcpc flag
m ssbs sb paca pacg dcpodp flagm
2 frint i8mm bti
Model name: Cortex-A710
Model: 0
Thread(s) per core: 1
Core(s) per socket: 3
Socket(s): 1
Stepping: r2p0
CPU(s) scaling MHz: 94%
CPU max MHz: 2496.0000
CPU min MHz: 633.6000
BogoMIPS: 38.40
Flags: fp asimd evtstrm aes pmull sha1
sha2 crc32 atomics fphp asimdhp
cpuid asimdrdm jscvt fcma lrcpc
dcpop sha3 sm3 sm4 asimddp sha51
2 asimdfhm dit uscat ilrcpc flag
m ssbs sb paca pacg dcpodp flagm
2 frint i8mm bti
Model name: -
Model: 0
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 1
CPU(s) scaling MHz: 26%
CPU max MHz: 2995.2000
CPU min MHz: 787.2000
Vulnerabilities:
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Spec store bypass: Mitigation; Speculative Store By
pass disabled via prctl
Spectre v1: Mitigation; __user pointer sanit
ization
Spectre v2: Vulnerable: Unprivileged eBPF en
abled
Srbds: Not affected
Tsx async abort: Not affected

[mqy]

build again and get the same result

Thanks

[ShadowPower]

I'm having the same problem with the Snapdragon 7+ Gen 2, so maybe it's a processor-specific issue.

[tcristo]

I have the same processor and same issue. Appears to be related to telling the compiler to optimize to the native processor. I use the following to work around it:

Find the directory the CMakelists.txt file is in, cd to that directory, and run the following line:

sed -i 's/add_compile_options(-mcpu=native)//g' CMakeLists.txt

Then make a build subdirectory, cd into it, and run cmake as follows:

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS='-mcpu=cortex-a35 -march=armv8.4a+dotprod' -S ../

Now run make.

You should now have a valid executable that will run without the illegal instruction problem.

[mqy]

sed -i 's/add_compile_options(-mcpu=native)//g' CMakeLists.txt

strange that:

1. LLAMA_NATIVE is OFF by default, add_compile_options(-march=native) should not be executed.
2. @adaaaaaa 's case: the main built with cmake works. make CFLAGS contains -mcpu=native but no -mfpu, that means $(UNAME_M) matches aarch64, but does not match armvX.

Related issue (previous fix) #1210

[adaaaaaa]

@tcristo
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS='-mcpu=cortex-a35 -march=armv8.4a+dotprod' -S ../
-- The C compiler identification is Clang 16.0.6
-- The CXX compiler identification is Clang 16.0.6
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /data/data/com.termux/files/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /data/data/com.termux/files/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /data/data/com.termux/files/usr/bin/git (found version "2.41.0")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- CMAKE_SYSTEM_PROCESSOR: aarch64
-- ARM detected
-- Configuring done (2.1s)
-- Generating done (0.1s)
-- Build files have been written to: /data/data/com.termux/files/home/llama.cpp/build

./main
main: build = 799 (dfd9fce)
main: seed = 1688670440
llama.cpp: loading model from models/7B/ggml-model.bin
llama_model_load_internal: format = ggjt v3 (latest)
llama_model_load_internal: n_vocab = 32000
llama_model_load_internal: n_ctx = 512
llama_model_load_internal: n_embd = 4096
llama_model_load_internal: n_mult = 256
llama_model_load_internal: n_head = 32
llama_model_load_internal: n_layer = 32
llama_model_load_internal: n_rot = 128
llama_model_load_internal: ftype = 3 (mostly Q4_1)
llama_model_load_internal: n_ff = 11008
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size = 0.08 MB
zsh: illegal hardware instruction ./main

get something different now...🙃

[JianbangZ]

1. add_compile_option

I have the same issue, I have a snapdragon 8 gen 2, I believe it's armv9a, same as 7 gen 2. May I know why you tell it's "mcpu=cortex-a35 -march=armv8.4a+dotprod?" a35 is quite old, and shouldn't be something like -march=armv9a?

[paviko]

Problem solved for new Snapdragon (armv9a) thanks to this thread:
#402

Find the directory the CMakelists.txt file is in (should be main dir) and run the following line:

sed -i 's/add_compile_options(-mcpu=native)/add_compile_options(-mcpu=native+nosve)/g' CMakeLists.txt

Snapdragon 8 gen 2 gives me 110ms per token for 7B Q4_K_S and -t 5

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.