Skip to content

Commit

Permalink
Update llmServer.py for backgroud model loading
Browse files Browse the repository at this point in the history
  • Loading branch information
seokho-son authored Feb 27, 2024
1 parent f4d4116 commit 9b140ed
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion scripts/usecases/llm/llmServer.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import uvicorn
# Correcting the import based on your initial code snippet
from langchain_community.llms import VLLM
import asyncio # Import asyncio for asynchronous task management

app = FastAPI()
port = 5001
Expand All @@ -26,7 +27,7 @@ async def load_model():

@app.on_event("startup")
async def startup_event():
await load_model()
asyncio.create_task(load_model()) # Schedule load_model to run as a background task

@app.get("/status")
def get_status():
Expand Down

0 comments on commit 9b140ed

Please sign in to comment.