Bugfix: Fix broken: UnicodeDecodeError: 'utf-8' codec can't decode

[riverzhou]

No description provided.

[CyberTimon]

Doesn't this just remove the error? Emoji's still don't work

[riverzhou]

Doesn't this just remove the error? Emoji's still don't work

Test passed in Chinese, not test Emoji.

[pjq]

Great to see the fix Chinese.

[jmtatsch]

Very funny indeed.
My vicuna prefers to answer me in Chinese.
With this fix at least it can without erroring out.

[MillionthOdin16]

Was this resolved Upstream? ggerganov/llama.cpp@aaf3b23

[abetlen]

@MillionthOdin16 I don't think so because I've had this issue on linux as well. I believe the issue is that utf-8 encodng is variable length and certain tokens are not valid utf-8 because they're just returned as bytes which may include partial utf-8 code points.

I think this needs some tests to ensure we're properly keeping track of the number of returned bytes.

[Niek]

Fixes #57

[abetlen]

@Niek can you confirm that this fixes the bug and gives the same result in streaming vs. regular mode. For example, compare streaming and regular mode for a completion that breaks in streaming mode with a fixed seed and temperature=0.

[Niek]

I just tested, for my own reference:

docker run --rm -it -v /path/to/models:/models -p8000:8000 python:3-buster bash
git clone -b river https://github.com/riverzhou/llama-cpp-python.git /app
cd /app
sed -i -e 's/[email protected]:/https:\/\/github.com\//' -e 's/.git$//' .gitmodules
git submodule update --init --recursive
python -m pip install --upgrade pip pytest cmake scikit-build setuptools fastapi sse_starlette uvicorn
python setup.py develop
HOST=0.0.0.0 MODEL=/models/ggml-vicuna-7b-4bit.bin python3 -m llama_cpp.server

With a prompt like show me 3 emojis, I no longer get an error, but it seems the message returned is empty instead. So doesn't look like a full fix.

[abetlen]

@Niek Can you try chaging for i in range(1,4): to for i in reversed(range(1,4)): so instead we're decoding the longest posible sequence? Not sure this would fix it either but worth a try.

[wujb123]

I change the code a littie bit, and it works:
_text = ''
try:
_text = text[start:].decode("utf-8")
except UnicodeDecodeError:
for i in range(-2,2): # changed to (-2,2)
try:
_text = text[start+i:].decode("utf-8") # changed to [start+i:]
break
except UnicodeDecodeError:
continue
yield {
"id": completion_id,
"object": "text_completion",
"created": created,
"model": self.model_path,
"choices": [
{
"text": _text,
"index": 0,
"logprobs": None,
"finish_reason": None,

[abetlen]

@riverzhou can you check if this bug is still occurs since #118 was merged?

[gjmulder]

@riverzhou update?

[riverzhou]

@riverzhou can you check if this bug is still occurs since #118 was merged?

Fine. It's OK now. Thanks.