Hosting LocalAI in cloud platforms such as AWS, Azure or GCP

[limcheekin]

I hosted the gpt4all-j model at AWS Lambda Function successfully (as it allowed memory size up to 10GB, container image size up to 10GB and up to 15 minutes timeout via function url) using the Python binding at https://pypi.org/project/gpt4all-j/. For low traffic app, I think it make sense to host the API in the cloud platforms, don't you think so?

We can use the AWS lambda function for experimental or development purpose also (which is low traffic by nature), so that it is pay for the usage, not the idle compute time.

From the cost perspective, I think it is overkill to host the LocalAI APIs on a dedicated server. I would like to hear from you if you found more cost effective way to host the LocalAI APIs.

Lastly, could we host the LocalAI APIs as AWS Lambda Function?

I hope to hear from you soon. Appreciate your sharing.

Thank you.

[simjak]

Hey, what is the performance on lambda? how long to you need to wait for the answer?

[simjak]

Have you tested AWS Fargate spot instances?

[limcheekin]

From my record, I deployed the open-llama-7b-400bt model via llama-cpp-python instead of LocalAI, it took 11 minutes to response for the first time included the model loading time. For the second request, it took about 60 to 90 seconds depends on the length of completion.

Deploying LocalAI to AWS Lambda is not that straight forward, I did some research on it but no time to work on it now, let's me know if you're interested, I will share the details here.

I didn't test for AWS Fargate. Kindly share your experience here if you did.

Thanks.

[simjak]

Hey, I haven't yet deployed on Fargate, still evaluating cloud providers, hardware setup and cost CPU vs GPU.
Not sure yet what is the best way here :)

[limcheekin]

Appreciate if you could share your findings and experiences here after you figured it out. I deployed to AWS Lambda mainly for taking advantage of the free tier plan.