RAG Agent stop tokens for multimode behaviour?

[jboero]

Hi has anyone put thought into handling different stop tokens for multimode models and RAG agent processes? It looks like there is already a framework for is_control_token() but not much is being done with it. This only does a continue instead of handling different types of control tokens.

llama.cpp/src/llama.cpp

Line 15732 in 6b2a849

if (cpt == 0 || cpt == 0xFFFD || flags.is_control) {

For example:

1. <URL>: pause completion and fetch the URL into context before continuing.
2. <SEARCH>: pause completion and attempt a web search.
3. <CALC>: pause completion to let math be calcluated via something like bc.
4. <COMPILE>: pause completion to try compiling code identified in Markdown tags.
5. <SCRIPT>: pause completion to run a script or containerized process.
6. <IMG>: pause completion to generate an image (if supported by model).

I see a lot more models recently supporting agents or external tooling. Arcee for example.
https://www.marktechpost.com/2024/07/06/arcee-ai-introduces-arcee-agent-a-cutting-edge-7b-parameter-language-model-specifically-designed-for-function-calling-and-tool-use/

Obviously with security implications this should be disabled by default and proper warnings to admins but the possibilities are interesting and could support more GPT4o-like behaviour.