llama : restore prefix space in llama tokenizer

[cebtenzzre]

I noticed a regression in models that do not use special tokens caused by #3538 (ref):

Command: build/bin/main -t 16 -m /tmp/dolphin-llama2-7b.Q4_0.new.gguf -n 2048 --ignore-eos -p 'The quick brown fox' --seed 1699464025

Before this PR (with a leading space, GitHub doesn't show it):

The quick brown fox jumps over the lazy dog is a well-known English nursery rhyme or children's song.➖

This nursery rhyme has been popular since 1847, when it was first published in the Monthly Magazine of London. Since then, it has undergone various changes and adaptations, but its essence remains the same. The rhyme tells the story of a quick brown fox who jumps over the lazy dog to avoid being caught for stealing some cheese.

After this PR (no leading space):

The quick brown fox jumps over the wall (the last line of the poem) and is on the other side.nahmáo Often used as a verb, němäo is a term derived from the Sanskrit word nirvana, meaning "spirit" or "consciousness." The concept of the spirit or consciousness can be found in many cultures and religions throughout history. In Buddhism, for instance, the term "nirvana" refers to the state of perfect spiritual awakening that is the goal of practice. Similarly, in Christianity, the Holy Spirit is considered a divine spirit that guides and supports believers.

It's just a random LLaMA-2 model that I use for testing, but its output quality is clearly reduced significantly by this change. I believe this applies to all other llama models that do not use special tokens to frame the prompt.

This is surprising behavior for downstream users - basic instruction-tuned (e.g. Vicuna, Alpaca) or writing-tuned models should just work, and I don't think anyone is adding the space that is currently necessary to match HF transformers. As mentioned here, there is some nuance in what transformers does that makes their tokenizer cooperate with both prompt formats, but I don't fully understand their implementation yet.

(Whether it really makes sense for HF transformers to put a leading space before the first ### Instruction: or USER: in the context of a multi-turn chat is another question, but that's how these models are being trained - it's done regardless of 'legacy' being True or False. AFAIK, there are models trained on Alpaca-style prompts that haven't seen multi-turn chats, and won't recognize ### Instruction: without the leading space.)

To be consistent, we should also honor add_prefix_space in the GPT2-style BPE tokenizer (cc @goerch). It appears to be true for Falcon and false for MPT. Right now, it seems that we don't store that in the GGUF at all.

[KerfuffleV2]

To be consistent, we should also honor add_prefix_space in the GPT2-style BPE tokenizer

That's from tokenizer_config.json? Can be added to the same GGUF python vocab.py logic in #4040 (note I linked that because it also includes a gguf-py change to the logic for that section)

[cebtenzzre]

I took shibe2's suggestion and made the space prefix conditional, but based on the position of the text relative to special tokens instead of based on whether special token processing is enabled. It seems to work fine without special tokens, but I haven't tested with special tokens.

[shibe2]

I will test it with a model that I already have. Someone else can also test it with a model that is known to perform better without extra spaces.

[shibe2]

I tested main executable. When the prompt begins with special token, tokenization result seems to be the same before and after the change. Inference works as expected.

[WeirdConstructor]

A question, seeing that the space is always added implicitly. Does that mean I can't tokenize partial prompts or in batches anymore? With the recent change in the batch and sequence API, separate tokenization of the system prompt and the sequences might be a thing that is going to happen. Sometimes you want to tokenize stuff that is added later and append it to some sequence.

[shibe2]

Does that mean I can't tokenize partial prompts or in batches anymore?

You can tokenize any string, full prompt or a part of it, but since #2810, a space is inserted into each non-empty string. This means that concatenation of tokenized parts of a prompt is not equivalent to tokenization of full prompt.

tokenize(x+y) != tokenize(x)+tokenize(y)

If you want them to be equivalent, you have to work around this behavior. For example:

llama.cpp/examples/infill/infill.cpp

Lines 236 to 247 in b12fa0d

	bool suff_rm_leading_spc = params.escape;
	if (suff_rm_leading_spc && params.input_suffix.find_first_of(" ") == 0 && params.input_suffix.size() > 1) {
	params.input_suffix.erase(0, 1);
	suff_rm_leading_spc = false;
	}
	std::vector<llama_token> embd_inp;
	std::vector<llama_token> inp_pfx = ::llama_tokenize(ctx, params.input_prefix, false);
	std::vector<llama_token> inp_sfx = ::llama_tokenize(ctx, params.input_suffix, false);
	const int space_token = 29871;
	if (suff_rm_leading_spc && inp_sfx[0] == space_token) {
	inp_sfx.erase(inp_sfx.begin());
	}

or #2810 (comment). Though this may not work for all models. I proposed a general solution in #3664.

This only affects models that use SentencePiece tokenizer (LLAMA_VOCAB_TYPE_SPM).

[WeirdConstructor]

If you want them to be equivalent, you have to work around this behavior. For example:

Thanks, I suspected as much. So I will need to do something like your workaround for now. The problem is, that I build prompts dynamically and attach as sequences in the KV cache. Leading to weird completion differences depending on where/when I call llama_tokenize().
Given that you have to insert BOS tokens at the beginning of prompts for some models, I expected this to behave similarly. Maybe there should be a llama_tokenize_start_prompt() and llama_tokenize_part(). Also instead of another bool flag in the arguments.
Your #3664 issue describes that well.

[KerfuffleV2]

@WeirdConstructor Sounds like you might want something like token healing: https://github.com/guidance-ai/guidance/blob/main/notebooks/token_healing.ipynb

[WeirdConstructor]

@WeirdConstructor Sounds like you might want something like token healing:

Ah yes, I am aware that tokens span multiple characters and splitting strings is not going to result in the same token vectors. Thanks for the heads up though.
It's just that llama_tokenize() inserting extra characters to the text is not helping with staying in control of the tokens I feed to the model.

[shibe2]

It's not difficult to split text on token boundaries, such that concatenation of token sequences would work (except for the issue with compulsory space). For example, I checked that in models that I use, newline (LF) character does not stick to any other tokens, so splitting the text into lines works. In general, conversational prompt formats don't allow tokens to contain pieces of different messages.

[KerfuffleV2]

@cebtenzzre I hate to say it, but it seems this broke stuff.

Testing with this model: https://huggingface.co/NousResearch/Nous-Capybara-34B/tree/main

Quantized versions: https://huggingface.co/TheBloke/Nous-Capybara-34B-GGUF

Comparing with these two prompts: USER: Do the thing and \nUSER: Do the thing (an actual newline in the second one):

>>> import sys
>>> sys.path.insert(0, '/path/Nous-Capybara-34B')
>>> import tokenization_yi as yt
>>> a = yt.YiTokenizer('/path/Nous-Capybara-34B/tokenizer.model', add_bos_token=False)
>>> [(x, a._convert_token_to_id(x)) for x in a._tokenize('USER: Do the thing')]
[('USER', 27323), (':', 59601), ('▁Do', 2994), ('▁the', 567), ('▁thing', 1919)]
>>> [(x, a._convert_token_to_id(x)) for x in a._tokenize('\nUSER: Do the thing')]
[('\n', 144), ('USER', 27323), (':', 59601), ('▁Do', 2994), ('▁the', 567), ('▁thing', 1919)]

Now testing with the tokenize example - note I had to fix:

// const bool add_bos = true;
const bool add_bos = llama_should_add_bos_token(model);

Here's what tokenizing looked like without this change:

 27323 -> 'USER'
 59601 -> ':'
  2994 -> ' Do'
   567 -> ' the'
  1919 -> ' thing'

and

   144 -> '
'
 27323 -> 'USER'
 59601 -> ':'
  2994 -> ' Do'
   567 -> ' the'
  1919 -> ' thing'

So this is as expected and matches the Python Yi tokenizer.

Now with this PR applied:

  2134 -> ' US'
  1471 -> 'ER'
 59601 -> ':'
  2994 -> ' Do'
   567 -> ' the'
  1919 -> ' thing

and

   59568 -> ' '
   144 -> '
'
 27323 -> 'USER'
 59601 -> ':'
  2994 -> ' Do'
   567 -> ' the'
  1919 -> ' thing'

The tokenize example sets special to true when tokenizing. With this PR, the results are identical regardless of what special is set to.

However, with the pre-PR version, it actually makes a difference. Tokenizing without this PR and special=false:

  2134 -> ' US'
  1471 -> 'ER'
 59601 -> ':'
  2994 -> ' Do'
   567 -> ' the'
  1919 -> ' thing'

and

 59568 -> ' '
   144 -> '
'
 27323 -> 'USER'
 59601 -> ':'
  2994 -> ' Do'
   567 -> ' the'
  1919 -> ' thing'

So basically the same as with this PR.

Anyway, it makes a huge difference in the quality of the response at least with the model I used as an example. The results are basically garbage with USER split into two tokens.

[shibe2]

@KerfuffleV2 Did you use --escape? If yes, try older version without it. I think, space insertion here is a deliberate decision. Nowadays, that decision does not look good, which I reported in #3664.

[KerfuffleV2]

@shibe2

Did you use --escape?

What do you mean? The tokenize example doesn't support anything other than --ids (to only print token ids)

When actually calling tokenize to produce this output, I just used the shell's input stuff to insert a literal newline in the prompt string. No need for escaping or whatever. (In zsh and bash you can hit ^V^J to do that.)

[shibe2]

@KerfuffleV2 Sorry, I didn't read your comment carefully. What I meant is that main without --escape might give same poor response before this change.

Anyway, it makes a huge difference in the quality of the response at least with the model I used as an example. The results are basically garbage with USER split into two tokens.

[KerfuffleV2]

What I meant is that main without --escape

I don't really understand. Are you saying --escape will have an effect other than interpreting escape codes in this case? Or are you thinking I did something like -p '\nUSER: Do the thing? (I didn't, I inserted a literal newline in the prompt text via the shell.)

[shibe2]

It just so happened that between #3538 and #4081, --escape could be used to control insertion of space in LLAMA_VOCAB_TYPE_SPM path.

[KerfuffleV2]

I still don't really know what you mean since #3538 doesn't appear to add any behavior based on --escape and #4081 doesn't remove that behavior, and it also doesn't seem to exist in the current version. I might have missed something looking at the code. Can you link to the exact commit you're saying did that?

[shibe2]

Now looking at the code I don't see it either. Perhaps, I confabulated it. I assumed that --escape is needed to enable processing of special tokens and was adding it to the command when testing prompt formats with special tokens. But it appears to be always on, which is not good, IMHO.

[KerfuffleV2]

@cebtenzzre Just wanted to make sure you saw this since there was a bunch of other discussion in the comments: #4081 (comment)

[cebtenzzre]

It seems like this PR was also significant for Mistral-Instruct. [INST] and [/INST] do not appear to be special tokens, so this PR restored the expected behavior of that model.

Without this change (using the Nomic Vulkan backend):

[INST] Write a brief, two-paragraph summary of the history of Unix. [/INST] Unix was first developed as a means of teaching computer science at the Bell Laboratories in 1968 by William H. Gates, who's contributions to Unix were significant and instrumental for its creation. Gates made his contributions free, that Laboratories will also be make for its history instrumental: Thomas H. Gates did a lot of work on the creation of the operating system. He developed an innovative version of Unix called: Unix, by William H. Gates himself. Gates' contributions did not create the first development of Unix until that his multi-user development of Unix was a major milestone in his career. The operating system he did not develop and named Unix was created from scratch by Garth' S. Thompson in 1968, at the urging of William H. Gates III, who was working on creating it at the time. In 1973, the creator of the Unix operating system, William H. Ritchie, made his contributions free, to the development of this, the early version of the Linux kernel was created by H. He was then instrumental in the development of Unix's early version of the same, William H. Bell at the time of its creation (1968). In 1973, at the time of the kernel's development, that is the most important Unix milestone, Ritchie, its contributions did a lot of work on creating a new Linux version that was made by his contributions. He released the first version of Unix, the Creation of the operating system, in 1973, in an early-development stage.

With this change:

[INST] Write a brief, two-paragraph summary of the history of Unix. [/INST] Unix was first developed in the late 1960s and early 1970s by Ken Thompson and Dennis Ritchie at Bell Laboratories, where Ken worked on developing the operating system for research purposes, while Dennis created the multuser and network support systems with the intention of providing a secure, reliable method to share filesystem resources among multiple users. Originally, Unix was just used as an experimental version of time-sharing by multiple developers in the 1980s, but its popularity led to it being used as a basis for multiuser and network development efforts for much of its history. The system was first introduced into the operating system in the early 1990s when it became clear that Unix was not just in the 1970s, but also in much of its workings through the 1990s. In the 2000s, Dennis was able to access Unix’s source code in his late 1970s was able to help him and Dennis with their multiuser and network development efforts. Since then, Unix has helped numerous developers by creating it as a multuser system in much of its workings and sharing history among them, making it possible to use in the 2010s when Ken Thompson and Dennis were able to support the production of Unix. When the original operating system was given their network development efforts in much of its file-sharing legacy, the team was able to develop the operating system for 3960-2470, but it was also able to use this new version during the 2010s when Dennis was working on developing a network of computer systems and file sharing networks at Bell Laboratories, the team was not just in much of their workings in Unix, but also in the operating system that was being given to Ken