-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LLM reporting Middleware that tracks token counts for AI APIs #6338
base: master
Are you sure you want to change the base?
Conversation
… estimates others)
API Changes --- prev.txt 2024-06-11 05:38:00.319166295 +0000
+++ current.txt 2024-06-11 05:37:57.343136072 +0000
@@ -6859,6 +6859,9 @@
GraphQLRequest
GraphQLIsWebSocketUpgrade
OASOperation
+ LLMReport_Model
+ LLMReport_NumRequestTokens
+ LLMReport_Estimate
// CacheOptions holds cache options required for cache writer middleware.
CacheOptions
@@ -8790,6 +8793,16 @@
func (l *LDAPStorageHandler) SetRollingWindow(keyName string, per int64, val string, pipeline bool) (int, []interface{})
+type LLMReport struct {
+ *BaseMiddleware
+}
+
+func (sa *LLMReport) EnabledForSpec() bool
+
+func (sa *LLMReport) Name() string
+
+func (sa *LLMReport) ProcessRequest(w http.ResponseWriter, r *http.Request, _ interface{}) (error, int)
+
type LogMessageEventHandler struct {
Gw *Gateway `json:"-"`
// Has unexported fields. |
PR Reviewer Guide 🔍
|
PR Code Suggestions ✨
|
💥 CI tests failed 🙈git-statediff --git a/gateway/mw_llm_reporter.go b/gateway/mw_llm_reporter.go
index 305ee78..bff0c82 100644
--- a/gateway/mw_llm_reporter.go
+++ b/gateway/mw_llm_reporter.go
@@ -8,8 +8,9 @@ import (
"os"
"strings"
- "github.com/TykTechnologies/tyk/ctx"
"github.com/pkoukk/tiktoken-go"
+
+ "github.com/TykTechnologies/tyk/ctx"
)
type msgObject struct { Please look at the run or in the Checks tab. |
💥 CI tests failed 🙈git-statediff --git a/gateway/mw_llm_reporter.go b/gateway/mw_llm_reporter.go
index 8336ca2..413f9ce 100644
--- a/gateway/mw_llm_reporter.go
+++ b/gateway/mw_llm_reporter.go
@@ -8,8 +8,9 @@ import (
"os"
"strings"
- "github.com/TykTechnologies/tyk/ctx"
"github.com/pkoukk/tiktoken-go"
+
+ "github.com/TykTechnologies/tyk/ctx"
)
type msgObject struct { Please look at the run or in the Checks tab. |
1 similar comment
💥 CI tests failed 🙈git-statediff --git a/gateway/mw_llm_reporter.go b/gateway/mw_llm_reporter.go
index 8336ca2..413f9ce 100644
--- a/gateway/mw_llm_reporter.go
+++ b/gateway/mw_llm_reporter.go
@@ -8,8 +8,9 @@ import (
"os"
"strings"
- "github.com/TykTechnologies/tyk/ctx"
"github.com/pkoukk/tiktoken-go"
+
+ "github.com/TykTechnologies/tyk/ctx"
)
type msgObject struct { Please look at the run or in the Checks tab. |
@lonelycode all LLMs this days return amount of spent tokens for request and response, so maybe instead of calculating it before request, calculate it after response? And after update the rates? |
User description
Adds a new middleware that uses tiktoken to count AI message tokens (support OpenAI and Anthropic)
Description
mw_llm_reporter.go
middlewarellm
LLMReporter
middleware willctx.LLMReport_Model
,ctx.LLMReport_NumTokens
, andctx.LLMReport_Estimate
.RecordHit
function to store the data in an an alytics record (as this is part of Tyk Pump I did not extend the pump record)Motivation and Context
We've had multiple requests for this kind of reporting and it is relatively easy to add to our analytics, it is also something our competitors already do and will allow us to have a matching feature.
How This Has Been Tested
Types of changes
Checklist
PR Type
Enhancement, Dependencies
Description
LLMReport
to track token counts for AI APIs using the tiktoken library.Changes walkthrough 📝
ctx.go
Add constants for LLM report context values.
ctx/ctx.go
api.go
Add functions to retrieve LLM report data from context.
gateway/api.go
estimate from request context.
api_loader.go
Enable LLMReport middleware in API processing.
gateway/api_loader.go
mw_llm_reporter.go
Implement LLMReport middleware for token counting.
gateway/mw_llm_reporter.go
tokens, and set context values.
using tiktoken library.
quickstart.json
Update quickstart API configuration for LLM reporting.
apps/quickstart.json
llm
tag to the quickstart API configuration.do_not_track
to false.go.mod
Add dependencies for LLM reporting middleware.
go.mod
tiktoken-go
andregexp2
dependencies.go.sum
Update go.sum with new dependencies.
go.sum