Automating llama.cpp

[javiagu13]

Dear all,

I am currently using llama for research purposes and I would like to know if there is any way to make kind of an "API". What I want to do is basically have a list of prompts as I would have for chatGPT and be able to send them and get the responses back in strings for further analysis. I work in the medical field so the main idea is sending prompts for different medical tasks such as triage or disease related prompts and get all the results for further analysis.

Thanks to all of you for building this amazing project!

[DannyDaemonic]

There's already a library. You can just make libllama.so. I don't think anyone's set up a dll yet but there's no reason it couldn't be compiled into a dll.

Unless, I guess you're looking for a more high level api? You can actually just run main with your parameters and your prompt and pipe the answer to a file or read from stdout directly. You can even use --prompt-cache to speed up future runs. Just make sure you set a reasonable --n_predict value. The only thing really missing is for reverse prompts to work in non-interactive mode as well (#1032). Still, I imagine it will be a thing sooner than later.

I'd suggest you play around with main in non-interactive mode (the default mode) and see if you can make that work.