instinct.cpp is a toolkit for developing LLM-powered applications.
🚨 This project is under active development and has not reached to GA stage of first major release. See more at Roadmap section.
What instinct.cpp offer:
- Applications that are working out-of-box.
- Assistant API server: Agent service that is fully compatible with OpenAI's Assistant API.
- mini-assistant-api: Single binary for single node deployment with vector database and other dependencies bundled.
mighty-assistant-api: (WIP) A cloud native implementation that is highly scalable with distributed components and multi-tenant support.
- chat-agent: A CLI application that create knowledge index with your docs (PDF,TXT,MD,...) and launch an HTTP server that is fully compatible with OpenAI
ChatCompletion.
- Assistant API server: Agent service that is fully compatible with OpenAI's Assistant API.
- Frameworks to build LLM-based applications. Say it
langchain.cpp.- Integration for privacy-first LLM providers: Built-in support for Ollama and other OpenAI compatible API services like vllm, llama.cpp server, nitro and more.
- Building blocks for common application patterns like Chatbot, RAG, LLM Agent.
- Functional chaining components for composable LLM pipelines.
- Agent patterns: ReACT, OpenAI-based tool agent, LLMCompiler, ...
For built-in applications:
For library itself:
Complete project plan is tracked at Project kanban.
| Milestone | Features | DDL |
|---|---|---|
| v0.1.0 | Long-short memory, PDF/TXT/DOCX ingestor, Chain programing paradigm, RAG reference app doc-agent |
3.29 |
| v0.1.1 | Performance tuning, RAG evaluation, Function calling agent | 4.16 |
| v0.1.2 | OpenAI Assistant API initial implementation, single-binary reference app mini-assistant |
4.30 |
| v0.1.3 | * mini-assistant: tool calls with opensourced LLMs |
5.17 |
| v0.1.4 | * doc-agent : rerank model* mini-assistant: file-search tool support. |
|
| v0.1.5 | Overall optimization | 6.30 |
| v0.1.6 | code-interpreter in mini-assistant |
7.15 |
Contributions are welcomed! You can join discord server, or contact me via email.
This project could not be possible without following awesome projects.
- bshoshany-thread-pool
- base64
- chatllm.cpp
- concurrentqueue
- cpptrace
- corssguid
- cpp-httplib
- duckx
- DuckDB
- exprtk
- fmt
- fmtlog
- hash_library
- icu
- inja
- libcurl
- llama.cpp
- nlohmann_json
- protobuf
- pdfium
- reactiveplusplus
- tsl-ordered-map
- uniparser
And many thanks to the shared training checkpoints from:
Lists are sorted alphabetically.