Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REST] Reuse client/connections whenever possible #748

Closed
myedibleenso opened this issue Jan 10, 2024 · 0 comments · Fixed by #749
Closed

[REST] Reuse client/connections whenever possible #748

myedibleenso opened this issue Jan 10, 2024 · 0 comments · Fixed by #749
Assignees
Labels
bug Something isn't working Integration

Comments

@myedibleenso
Copy link
Collaborator

We make a series of proxied calls in our services. Creating a new client/client connection for each call to the same remote introduces latency (see https://stackoverflow.com/a/34491383). Whenever possible, we should reuse connections. One strategy is to utilize FastAPI's dependency injection which supports caching of dependencies: https://fastapi.tiangolo.com/tutorial/dependencies/sub-dependencies/

Relates to #747

@myedibleenso myedibleenso added bug Something isn't working Integration labels Jan 10, 2024
@myedibleenso myedibleenso added this to the [DARPA] Milestone 11 milestone Jan 10, 2024
@myedibleenso myedibleenso self-assigned this Jan 10, 2024
github-actions bot added a commit that referenced this issue Jan 16, 2024
## Summary of Changes

This PR introduces performance-related improvements.

- Avoids proxied calls to our remote deployment for the `llm_proxy`
service (see #746 )
- Moves from `requests` to `httpx` for asynchronous calls to other
services (see #747 )
- Reuses clients for improved performance via dependency injection (see
#748 )

### Related issues

- Resolves #746
- Resolves #747
- Resolves #748

---------

Co-authored-by: Justin <[email protected]> ea7e2b6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Integration
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant