[Question] [Optimization] async support?

[logan-markewich]

I noticed we are using fastapi to build micro services, but we aren't using async for anything?

For example, llamaindex has full async support on nearly every component. Yet all the methods are defined using synchronous methods.

We can greatly increase throughput but using async with fastapi

[kevinintel]

Thanks for reminding, we already use async

[logan-markewich]

@kevinintel for sure, but shouldn't the components be using async? For example for embeddings, we have await embed_model.aget_text_embedding() or with llms await llm.achat() or with vector stores vector_store.aquery() and so on

I don't see this being utilized

[lvliang-intel]

@logan-markewich，
we currently have async support in some microservices, but not all. We'll need to evaluate each case individually. Thank you for highlighting this issue; addressing it can significantly improve the performance of the entire pipeline.

You can find the async support on the LLM microservice.
https://github.com/opea-project/GenAIComps/blob/main/comps/llms/text-generation/tgi/llm.py