Skip to content

Latest commit

 

History

History
199 lines (147 loc) · 8.03 KB

README.md

File metadata and controls

199 lines (147 loc) · 8.03 KB
Raw

NOTICE: This is an experimental, unstable, incomplete project. Maybe give me a star so I continue working on it ;)

🌊 Welcome aboard the OverBott express, where innovation meets intuition and privacy races ahead at warp speed! 🚀🐢


Build Status Github Tag Discord

Get ready to dive into the digital future with OverBott, your ultimate sidekick for navigating the vast realms of AI-powered tools. Seamlessly blending with the leading tech pioneers like Ollama and OpenAI, OverBott propels your digital journey to new heights.

Features / Stack

  • Full Offline Capable / Privacy-first / Full data control (View/delete data, one click away)
  • Preset system - Create/Manage/Export/Import presets with your favorite workflows
  • Multiple LLM Providers (Offline & Online)
    • Locally Running Providers (Ollama, Localai, GPT4All, LlamaCPP)
    • Online Providers (OpenAI, MistralAI, Claude, Groq)
  • RAG (Retrieval Augmented Generation) over PDF
  • Local image generation with stable diffusion (supports HF models, pipelines, schedules)
  • Text-to-speech 27 Languages with OpenTTS Server

High-level Objectives

  • Experiment around the foundation of the app
  • Integrate with Ollama API [List/Pull/Delete/Chat/ChatCompletion]
  • Integrate with LocalAI/OpenAI
  • Implement Langchain (Create/manage Agents and Tools)
  • Video/Music Player [Local/Spotify/Youtube/Vimeo/...]
  • Mini Web browser (Allows agents to have access to vision+source in web-pages)
  • Preset System (Create/Manage system presets/profiles)
  • Voice System (OpenVoice TTS & STT/Elevenlabs)
  • Command System
    • Shortcuts for everything
    • Create in chat commands
      • /play [music]
      • /draw [image]
      • /search [browser]
      • /study [subject]

Research & Development

  • Should we build this desktop only?
    • Pros: Somewhat niche specific, mostly users/businesses with more performant machines.
    • Cons:
      • Cannot deploy the solution somewhere online for better performance.
      • Cannot re-use full code-base (electron-side) in case of going Mobile
  • Should we focus on RAG driven app only?
    • We should focus on SOMETHING, and RAG can be pretty broad.
      • We can focus on RAG for:
        • Q&A/Summarization
        • Question-Building (Study focused app)
  • What other directions can a project like this take?

Install

Clone the repo and install dependencies:

git clone --branch main https://github.com/splitpierre/overbott.git your-project-name
cd your-project-name
npm install

Having issues installing? See our debugging guide

Starting Development

Start the app in the dev environment:

npm start

Packaging for Production

To package apps for the local platform:

npm run package

Llama CPP (server setup)

git clone https://github.com/ggerganov/llama.cpp.git && cd llama.cpp
mkdir MODELS (Add guff Q4_0 models here)
# Add to .devops/server-vulkan.Dockerfile (or other images you need)
COPY MODELS /models
# Build docker image
docker build -t llama-cpp-vulkan:server -f .devops/server-vulkan.Dockerfile .
# Run and serve the docker image at port 4893 / Check change --device card and model name
# --n-gpu-layers should adjust according to video card VRAM available
# 8GBVRAM => 48 | 6GBVRAM => 32 | 4GBVRAM => 24 | 2GBVRAM => 12 ...

docker run -it --rm -v "$(pwd):/app:Z" --device /dev/dri/renderD128:/dev/dri/renderD128 --device /dev/dri/card0:/dev/dri/card0 -p 4893:4893 llama-cpp-vulkan:server -m "/models/Nous-Hermes-2-Mistral-7B-DPO.Q4_0.gguf"  --port 4893 --host 0.0.0.0 -n 2048 --n-gpu-layers 32
# Notice: At least on Vulkan, with AMD RX 570, only Q4_0 < 4gb models can be used.

Unstructured API

docker pull downloads.unstructured.io/unstructured-io/unstructured-api:latest
docker run -p 8000:8000 -d --rm --name unstructured-api downloads.unstructured.io/unstructured-io/unstructured-api:latest --port 8000 --host 0.0.0.0

OpenTTS

API Swagger -> localhost:5500/openapi

docker run -it -p 5500:5500 synesthesiam/opentts:en --no-espeak

docker-diffusers-api

# Serve on 8077 (with/without gpu)
docker run --gpus all -p 8077:8000 -e HF_AUTH_TOKEN=$HF_AUTH_TOKEN gadicc/diffusers-api
docker run -p 8077:8000 -e HF_AUTH_TOKEN=$HF_AUTH_TOKEN gadicc/diffusers-api

Donations

Donations will ensure the following:

  • 🔨 Long term maintenance of the project
  • 🛣 Progress on the roadmap
  • 🐛 Quick responses to bug reports and help requests

Troubleshooting

  • Issue: After app install, making GPU driver changes might break Electron/App GPU associated cache cache.
    • ERROR:shared_context_state.cc(83)] Skia shader compilation error
    • find ~/.config/[Electron]/ -name "GPUCache" | xargs rm -rf
    • replace [Electron] to repo, electron and related caches to reset

Maintainers

Acknowledgements

  • CompVis, Stability AI, LAION and RunwayML for their incredible time, work and efforts in creating Stable Diffusion, and no less so, their decision to release it publicly with an open source license.
  • HuggingFace - for their passion and inspiration for making machine learning more accessibe to developers, and in particular, their Diffusers library.

License

MIT © OverBott