Enable special token support for exllamav2

[JohanAR]

Checklist:

• I have read the Contributing guidelines.

[Ph0rk0z]

should it also use 'skip_special_tokens', ?

[JohanAR]

should it also use 'skip_special_tokens', ?

Not sure.. skip_special_tokens appears to be a setting for the Transformers tokenizer, but it seems like this might be the same as setting decode_special_tokens=False for exllamav2

[JohanAR]

I updated the commit to add this setting and it appears to be working as expected.

Though with the only models I have that use special tokens (ChatML ending reply with <|im_end|>) I think at least in the notepad tab one would want skip_special_tokens=False for multi-turn conversations, but it seems like it defaults to True for all models. Not sure about the other tabs.

Also I'm not entirely sure about the decode function which doesn't take a state parameter. If I hard code it to False there I think the notebook Tokens sub-tab wrote empty strings instead of special tokens, so True appears to be better. No idea if it affects other things.

[oobabooga]

Thanks for the PR. The state parameter goes to the generation functions, which can pass state['skip_special_tokens'] to the decode function as you did. I think that your implementation is correct and no further changes are needed.