Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LiteLLM Minor Fixes & Improvements (09/18/2024) #5772

Merged
merged 19 commits into from
Sep 19, 2024
Merged

LiteLLM Minor Fixes & Improvements (09/18/2024) #5772

merged 19 commits into from
Sep 19, 2024

Conversation

krrishdholakia
Copy link
Contributor

@krrishdholakia krrishdholakia commented Sep 18, 2024 •
edited
Loading

Title

  • feat(cost_calculator.py): support fireworks ai cost tracking
  • build(docker-compose.yml): add lines for mounting config.yaml to docker compose
  • docs(input.md): update docs to clarify litellm supports content as a list of dictionaries
  • fix(image_handling.py): follow image url redirects
  • fix(router.py): Fix model key/base leak in error message

Relevant issues

Fixes #4570
Closes #5739
Fixes #5755
Fixes #5763
Fixes #5762

Type

🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test

Changes

[REQUIRED] Testing - Attach a screenshot of any new tests passing locall

If UI changes, send a screenshot/GIF of working UI fixes

@vercel Vercel
Copy link

vercel bot commented Sep 18, 2024 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
litellm ✅ Ready (Inspect) Visit Preview 💬 Add feedback Sep 19, 2024 4:00pm

github-advanced-security[bot]
github-advanced-security bot found potential problems Sep 18, 2024
View reviewed changes
litellm/llms/fireworks_ai/cost_calculator.py
model_name = model_name.lower()

# Check for MoE models in the form <number>x<number>b
moe_match = re.search(r"(\d+)x(\d+)b", model_name)

Check failure

Code scanning / CodeQL

Polynomial regular expression used on uncontrolled data High

This
regular expression
Loading
that depends on a
user-provided value
Loading
may run slow on strings with many repetitions of '0'.
This
regular expression
Loading
that depends on a
user-provided value
Loading
may run slow on strings with many repetitions of '0'.
This
regular expression
Loading
that depends on a
user-provided value
Loading
may run slow on strings with many repetitions of '0'.
This
regular expression
Loading
that depends on a
user-provided value
Loading
may run slow on strings with many repetitions of '0'.
litellm/llms/fireworks_ai/cost_calculator.py
return "fireworks-ai-56b-to-176b"

# Check for standard models in the form <number>b
re_params_match = re.search(r"(\d+)b", model_name)

Check failure

Code scanning / CodeQL

Polynomial regular expression used on uncontrolled data High

This
regular expression
Loading
that depends on a
user-provided value
Loading
may run slow on strings with many repetitions of '0'.
This
regular expression
Loading
that depends on a
user-provided value
Loading
may run slow on strings with many repetitions of '0'.
This
regular expression
Loading
that depends on a
user-provided value
Loading
may run slow on strings with many repetitions of '0'.
This
regular expression
Loading
that depends on a
user-provided value
Loading
may run slow on strings with many repetitions of '0'.
litellm/tests/test_completion_cost.py
model="fireworks_ai/mixtral-8x7b-instruct", messages=messages
) # works fine

print(resp)

Check failure

Code scanning / CodeQL

Clear-text logging of sensitive information High test

This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expression logs
sensitive data (secret)
Loading
as clear text.
This expressi

Copilot Autofix AI 3 days ago

To fix the problem, we should avoid logging the entire resp object directly. Instead, we can log only non-sensitive parts of the response or use a logging mechanism that redacts sensitive information.

The best way to fix this without changing existing functionality is to:

  1. Identify and log only the non-sensitive parts of the resp object.
  2. Use a custom logging function that redacts or omits sensitive information.
Suggested changeset 1
litellm/tests/test_completion_cost.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/litellm/tests/test_completion_cost.py b/litellm/tests/test_completion_cost.py
--- a/litellm/tests/test_completion_cost.py
+++ b/litellm/tests/test_completion_cost.py
@@ -1275,3 +1275,7 @@
 
-    print(resp)
+    # Log only non-sensitive parts of the response
+    if hasattr(resp, 'non_sensitive_data'):
+        print(f"Non-sensitive response data: {resp.non_sensitive_data}")
+    else:
+        print("Response received, but contains sensitive data.")
     cost = completion_cost(completion_response=resp)
EOF
@@ -1275,3 +1275,7 @@

print(resp)
# Log only non-sensitive parts of the response
if hasattr(resp, 'non_sensitive_data'):
print(f"Non-sensitive response data: {resp.non_sensitive_data}")
else:
print("Response received, but contains sensitive data.")
cost = completion_cost(completion_response=resp)
Copilot is powered by AI and may make mistakes. Always verify output.
Unable to commit as this autofix suggestion is now outdated
Positive Feedback
Negative Feedback

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Please select one or more of the options
github-advanced-security[bot]
github-advanced-security bot found potential problems Sep 18, 2024
View reviewed changes
litellm/_redis.py Fixed Show fixed Hide fixed
litellm/_redis.py Fixed Show fixed Hide fixed
@dbczumar @krrishdholakia @nirga
Databricks Integration: Integrate Databricks SDK as optional mechanis…
c96fccb 
…m for fetching API base and token, if unspecified (#5746)

* LiteLLM Minor Fixes & Improvements (09/16/2024)  (#5723)

* coverage (#5713)

Signed-off-by: dbczumar <[email protected]>

* Move (#5714)

Signed-off-by: dbczumar <[email protected]>

* fix(litellm_logging.py): fix logging client re-init (#5710)

Fixes #5695

* fix(presidio.py): Fix logging_hook response and add support for additional presidio variables in guardrails config

Fixes #5682

* feat(o1_handler.py): fake streaming for openai o1 models

Fixes #5694

* docs: deprecated traceloop integration in favor of native otel (#5249)

* fix: fix linting errors

* fix: fix linting errors

* fix(main.py): fix o1 import

---------

Signed-off-by: dbczumar <[email protected]>
Co-authored-by: Corey Zumar <[email protected]>
Co-authored-by: Nir Gazit <[email protected]>

* feat(spend_management_endpoints.py): expose `/global/spend/refresh` endpoint for updating material view (#5730)

* feat(spend_management_endpoints.py): expose `/global/spend/refresh` endpoint for updating material view

Supports having `MonthlyGlobalSpend` view be a material view, and exposes an endpoint to refresh it

* fix(custom_logger.py): reset calltype

* fix: fix linting errors

* fix: fix linting error

* fix

Signed-off-by: dbczumar <[email protected]>

* fix: fix import

* Fix

Signed-off-by: dbczumar <[email protected]>

* fix

Signed-off-by: dbczumar <[email protected]>

* DB test

Signed-off-by: dbczumar <[email protected]>

* Coverage

Signed-off-by: dbczumar <[email protected]>

* progress

Signed-off-by: dbczumar <[email protected]>

* fix

Signed-off-by: dbczumar <[email protected]>

* fix

Signed-off-by: dbczumar <[email protected]>

* fix

Signed-off-by: dbczumar <[email protected]>

* fix test name

Signed-off-by: dbczumar <[email protected]>

---------

Signed-off-by: dbczumar <[email protected]>
Co-authored-by: Krish Dholakia <[email protected]>
Co-authored-by: Nir Gazit <[email protected]>
@krrishdholakia @github-advanced-security
Apply code scanning fix for clear-text logging of sensitive information
4ddc181 
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
@krrishdholakia krrishdholakia merged commit d46660e into main Sep 19, 2024
15 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@krrishdholakia @dbczumar