feat: Cache generated responses

[whiterabbit1983]

Closes #325

🚀 This description was created by Ellipsis for commit 0529b48

Summary:

Introduces caching for generated responses in the agents-api, including new utility functions, a caching decorator, a database migration for cache storage, and a new dependency.

Key points:

• Added get_cached_response and set_cached_response functions in /agents-api/agents_api/models/session/ for caching.
• Implemented cache decorator in /agents-api/agents_api/routers/sessions/session.py to enable caching for the generate method.
• Created migration script /agents-api/migrations/migrate_1716013793_session_cache.py for session_cache table.
• Added xxhash dependency in pyproject.toml for generating cache keys.

---

Generated with ❤️ by ellipsis.dev

---

🚀 This description was created by Ellipsis for commit ee589df

Summary:

Enhances the agents-api by introducing caching for generated responses, improving performance and efficiency.

Key points:

• Introduces caching for generated responses in the agents-api to enhance performance and efficiency.
• Added caching utility functions get_cached_response and set_cached_response in /agents-api/agents_api/models/session/
• Implemented cache decorator in /agents-api/agents_api/routers/sessions/session.py for the generate method
• Created migration script /agents-api/migrations/migrate_1716013793_session_cache.py for session_cache table
• Added xxhash dependency in pyproject.toml for cache keys

---

Generated with ❤️ by ellipsis.dev

[sweep-ai]

Sweep: PR Review

agents-api/agents_api/models/session/get_cached_response.py

Introduced a new function get_cached_response to query a session cache using a provided key, and added the necessary import for the cozo_query decorator.

Sweep Found These Issues

• The function get_cached_response does not handle the case where the key is not found in the session_cache, which could lead to unexpected behavior or errors.

julep/agents-api/agents_api/models/session/get_cached_response.py

Lines 5 to 11 in 0529b48

	def get_cached_response(key: str) -> tuple[str, dict]:
	query = """
	input[key] <- [[$key]]
	?[key, value] := input[key], *session_cache{key, value}
	"""
	return (query, {"key": key})


View Diff

---

agents-api/agents_api/models/session/set_cached_response.py

Introduced a new function set_cached_response to cache key-value pairs in a session_cache.

Sweep Found These Issues

• The function does not handle potential exceptions that may occur during the query execution, which could lead to unhandled errors.

julep/agents-api/agents_api/models/session/set_cached_response.py

Lines 5 to 16 in 0529b48

	def set_cached_response(key: str, value: dict) -> tuple[str, dict]:
	query = """
	?[key, value] <- [[$key, $value]]
	:insert session_cache {
	key => value
	}
	:returning
	"""
	return (query, {"key": key, "value": value})


View Diff

Potential Issues

Sweep isn't 100% sure if the following are issues or not but they may be worth taking a look at.

• The function does not validate the input parameters key and value, which could lead to unexpected behavior or security vulnerabilities if invalid data is passed.

julep/agents-api/agents_api/models/session/set_cached_response.py

Lines 5 to 16 in 0529b48

	def set_cached_response(key: str, value: dict) -> tuple[str, dict]:
	query = """
	?[key, value] <- [[$key, $value]]
	:insert session_cache {
	key => value
	}
	:returning
	"""
	return (query, {"key": key, "value": value})


View Diff
• The function does not validate the input parameters key and value, which could lead to unexpected behavior or security vulnerabilities if invalid data is passed.

julep/agents-api/agents_api/models/session/set_cached_response.py

Lines 5 to 16 in 0529b48

	def set_cached_response(key: str, value: dict) -> tuple[str, dict]:
	query = """
	?[key, value] <- [[$key, $value]]
	:insert session_cache {
	key => value
	}
	:returning
	"""
	return (query, {"key": key, "value": value})


View Diff

---

agents-api/agents_api/routers/sessions/session.py

The changes introduce a caching mechanism using the xxhash library and new functions to store and retrieve cached responses, applied to the generate method of the BaseSession class.

Potential Issues

Sweep isn't 100% sure if the following are issues or not but they may be worth taking a look at.

• The cache decorator assumes that the result returned by get_cached_response can be directly converted into a ChatCompletion object, which might not always be the case, potentially causing a ValueError.

julep/agents-api/agents_api/routers/sessions/session.py

Line 77 in 0529b48

return ChatCompletion(**result.iloc[0].to_dict())


View Diff

---

agents-api/migrations/migrate_1716013793_session_cache.py

The changes introduce a new migration script to create and remove a "session_cache" table with key-value pairs.

Sweep Found These Issues

• The SQL command for creating the "session_cache" table uses a non-standard syntax with :create and ::remove, which may not be recognized by the database engine.

julep/agents-api/migrations/migrate_1716013793_session_cache.py

Lines 9 to 16 in 0529b48

	:create session_cache {
	key: String,
	=>
	value: Json,
	}
	""",
	down="""
	::remove session_cache


View Diff

---

The following files were not reviewed because our filter identified them as typically non-human-readable or less important files (e.g., dist files, package.json, images). If this is an error, please let us know.

• agents-api/poetry.lock
• agents-api/pyproject.toml

[ellipsis-dev]

❌ Changes requested. Reviewed everything up to 0529b48 in 2 minutes and 9 seconds

More details

• Looked at 161 lines of code in 5 files
• Skipped 1 files when reviewing.
• Skipped posting 0 drafted comments based on config settings.

Workflow ID: wflow_ALt1hiVDIV7A4xm9

---

Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

[creatorrr]

Lgtm

[ellipsis-dev]

❌ Changes requested. Incremental review on 857d22f in 3 minutes and 34 seconds

More details

• Looked at 68 lines of code in 2 files
• Skipped 0 files when reviewing.
• Skipped posting 0 drafted comments based on config settings.

Workflow ID: wflow_ZLFLTaEafTaSQgCz

---

Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

[ellipsis-dev]

❌ Changes requested. Incremental review on 5e36e05 in 2 minutes and 2 seconds

More details

• Looked at 27 lines of code in 1 files
• Skipped 0 files when reviewing.
• Skipped posting 0 drafted comments based on config settings.

Workflow ID: wflow_gr0rbaliEjp36YPY

---

Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

[ellipsis-dev]

👍 Looks good to me! Incremental review on ee589df in 3 minutes and 48 seconds

More details

• Looked at 49 lines of code in 2 files
• Skipped 0 files when reviewing.
• Skipped posting 0 drafted comments based on config settings.

Workflow ID: wflow_h0z5NvAESTETn6uH

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.