-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
query-frontend: populate org_id in slow query logs #3177
Comments
We have a few different clients which interact with Thanos, which provide various headers which are useful for finding the source of queries (e.g. X-Grafana-User which contains the user name of a logged in Grafana user). Would it be useful if Thanos query-frontend took a list of potential header keys to try to extract an OrgId, then fallback through them until it finds one which isn’t empty, something like query It seems like it would be straightforward to add (and I'd be happy to contribute), but a lot depends how valuable other people find the slow query log. |
Sounds good to me ! |
Awesome. I'll have a look at getting a PR together for that. |
I think this will work as short term solution, but we should think how to define multitenancy in Thanos long term. (: |
The OrgId passed to Cortex is hard-coded to "fake":
thanos/cmd/thanos/query-frontend.go
Line 203 in 5aac477
When logs are produced to the slow query log, it results in logs which contain
org_id=fake
, e.g:level=info ts=2020-09-16T13:33:14.670646Z caller=frontend.go:206 org_id=fake msg="slow query detected" method=GET host=localhost:10902 path=/api/v1/query_range time_taken=890.912705ms ...
When using Cortex Query Frontend (which populates org_id based on the contents of the X-Scope-OrgID header), there was extra value in these logs. The org_id field is useful for identifying sources of problematic queries which take a long time to execute.
Does anyone have any thoughts on populating that org_id log field with the contents of a request header?
The text was updated successfully, but these errors were encountered: