Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[REST] Reuse client/connections whenever possible #748

Closed
myedibleenso opened this issue Jan 10, 2024 · 0 comments · Fixed by #749
Closed

[REST] Reuse client/connections whenever possible #748

myedibleenso opened this issue Jan 10, 2024 · 0 comments · Fixed by #749
Assignees
@myedibleenso
Labels
bug Something isn't working Integration
Milestone
[DARPA] Milestone 11

Comments

@myedibleenso
Copy link
Collaborator

We make a series of proxied calls in our services. Creating a new client/client connection for each call to the same remote introduces latency (see https://stackoverflow.com/a/34491383). Whenever possible, we should reuse connections. One strategy is to utilize FastAPI's dependency injection which supports caching of dependencies: https://fastapi.tiangolo.com/tutorial/dependencies/sub-dependencies/

Relates to #747
@myedibleenso myedibleenso added bug Something isn't working Integration labels Jan 10, 2024
@myedibleenso myedibleenso added this to the [DARPA] Milestone 11 milestone Jan 10, 2024
@myedibleenso myedibleenso self-assigned this Jan 10, 2024
@myedibleenso myedibleenso mentioned this issue Jan 10, 2024
Merged
github-actions bot added a commit that referenced this issue Jan 16, 2024
@github-actions
[REST] Performance improvements (async and client reuse) (#749)
3ba73e9 
## Summary of Changes

This PR introduces performance-related improvements.

- Avoids proxied calls to our remote deployment for the `llm_proxy`
service (see #746 )
- Moves from `requests` to `httpx` for asynchronous calls to other
services (see #747 )
- Reuses clients for improved performance via dependency injection (see
#748 )

### Related issues

- Resolves #746
- Resolves #747
- Resolves #748

---------

Co-authored-by: Justin <[email protected]> ea7e2b6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@myedibleenso myedibleenso

Labels
bug Something isn't working Integration
Projects
None yet
Milestone
[DARPA] Milestone 11
Development

Successfully merging a pull request may close this issue.

[REST] Performance improvements (async and client reuse) ml4ai/skema
1 participant
@myedibleenso