main: use seperate stream for control characters

[mofosyne]

Stab at this idea for ggerganov#6923

One thing you could do is print the special tokens out of band to file descriptor 3. Then if a shell script doesn't want them, it could either pass a flag to disable them, or simply say 3>/dev/null.

[mrdomino]

I am not 100% sure about the overall approach yet, but the implementation looks basically sound aside from the comments.

[mofosyne]

Fixed @mrdomino ready for you to test

[mrdomino]

stdout is buffered, so if you are going to be using it, you will need to make sure fflush(stdout) is called before every write to the token fd. Otherwise the tokens will not interleave correctly.

You could also switch to calling write on fd 1 and make the output fully unbuffered, but I think this would be suboptimal.

[mofosyne]

added fflush(stdout) before every fprintf() now

[jart]

Why wouldn't you flush after each print?

[mrdomino]

@jart my theory was that you'd want to take advantage of the buffering. If you don't care about buffering output then absolutely, or even setbuf(stdout, NULL) at the start of the program.

ETA: the other thing is that if you have anything else doing anything to stdout it can mess with the interleaving... it just seemed simpler to me to constrain it to where you were doing the token write

[mofosyne]

@mrdomino thanks for the command string you used to test.

This took a while, but I realised now that the C file open and bash are using the same file descriptor... so we really actually needed to check right at the start of the program to fully check if the file descriptor was already claimed by bash.

Anyway this is the full test process to prove that this is behaving as I was expecting, you may want to probe it a bit further, but I'm a bit more confident now that I know what I'm doing now here.

cmake -B build -DCMAKE_BUILD_TYPE=Debug
cmake --build build

echo "== Expect Control Token To Shared Console 3>&1 =="
./build/bin/main --hf-repo TheBloke/phi-2-GGUF --hf-file phi-2.Q6_K.gguf --grammar 'root ::= "yes" | "no"' --temp 0 -c 0 --no-display-prompt --log-disable -p "<|user|>
Say yes
<|assistant|>" 2>/dev/null 3>&1
echo
echo "== Expect No Control Token To Console because 3>/dev/null =="
./build/bin/main --hf-repo TheBloke/phi-2-GGUF --hf-file phi-2.Q6_K.gguf --grammar 'root ::= "yes" | "no"' --temp 0 -c 0 --no-display-prompt --log-disable -p "<|user|>
Say yes
<|assistant|>" 2>/dev/null 3>/dev/null
echo
echo "== Expect No Control Token To Console because 3>&- =="
./build/bin/main --hf-repo TheBloke/phi-2-GGUF --hf-file phi-2.Q6_K.gguf --grammar 'root ::= "yes" | "no"' --temp 0 -c 0 --no-display-prompt --log-disable -p "<|user|>
Say yes
<|assistant|>" 2>/dev/null 3>&-
echo
echo == Expect No Control Token To Console as we are still in grammar mode ==
./build/bin/main --hf-repo TheBloke/phi-2-GGUF --hf-file phi-2.Q6_K.gguf --grammar 'root ::= "yes" | "no"' --temp 0 -c 0 --no-display-prompt --log-disable -p "<|user|>
Say yes
<|assistant|>" 2>/dev/null
echo
echo == Expect Control Token To Console as we are in normal completion mode ==
./build/bin/main --hf-repo TheBloke/phi-2-GGUF --hf-file phi-2.Q6_K.gguf --temp 0 -c 0 --no-display-prompt --log-disable -p "<|user|>
Hi
<|assistant|>" 2>/dev/null 3>&1
echo

Which would result in this output

== Expect Control Token To Shared Console 3>&1 ==
yes<|endoftext|>
== Expect No Control Token To Console because 3>/dev/null ==
yes
== Expect No Control Token To Console because 3>&- ==
yes
== Expect No Control Token To Console as we are still in grammar mode ==
yes
== Expect Control Token To Console as we are in normal completion mode ==


I'm glad you found the information helpful. If you have any more questions or need further assistance, feel free to ask.

Best regards,
<|assistant|>
<|endoftext|>

[jart]

Why wouldn't you flush after each print?

[mrdomino]

extraneous ! here?

[mrdomino]

also I'm not sure why you'd have (void) before write but not fprintf

[mofosyne]

intentional or it breaks make command

(void) write(SPECIAL_FILENO, token_str.c_str(), token_str.length());

examples/main/main.cpp: In function ‘int main(int, char**)’:
examples/main/main.cpp:767:37: warning: ignoring return value of ‘ssize_t write(int, const void*, size_t)’ declared with attribute ‘warn_unused_result’ [-Wunused-result]
  767 |                         (void) write(SPECIAL_FILENO, token_str.c_str(), token_str.length());
      |                                ~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[mrdomino]

yep, apparently gcc and glibc have collaborated to do an extremely dumb thing. fine, keep it.

[mofosyne]

Context: https://bugs.llvm.org/show_bug.cgi?id=51228
Context: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66425

A possible appproach according to @mrdomino

#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wunused-result"
write(...);
#pragma GCC diagnostic pop

[mrdomino]

Looks good to me