-
Notifications
You must be signed in to change notification settings - Fork 514
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Increase default max size and add troubleshotting page (#1688)
* Increase default max size and add troubleshotting page * Changelog * Changelog * Apply suggestions from code review Co-authored-by: Kim Nylander <[email protected]> Co-authored-by: Kim Nylander <[email protected]>
- Loading branch information
1 parent
39110a8
commit fc83ff5
Showing
4 changed files
with
61 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
--- | ||
title: Response larger than the max | ||
weight: 477 | ||
--- | ||
|
||
# Response too large | ||
|
||
The error message will take a similar form to the following: | ||
|
||
``` | ||
500 Internal Server Error Body: response larger than the max (<size> vs <limit>) | ||
``` | ||
|
||
This error indicates that the response received or sent is too large. | ||
This can happen in multiple places, but it's most commonly seen in the query path, | ||
with messages between the querier and the query frontend. | ||
|
||
## Solutions | ||
|
||
### Tempo server (general) | ||
|
||
Tempo components communicate with each other via gRPC requests. | ||
To increase the maximum message size, you can increase the gRPC message size limit in the server block. | ||
|
||
```yaml | ||
server: | ||
grpc_server_max_recv_msg_size: <size> | ||
grpc_server_max_send_msg_size: <size> | ||
``` | ||
The server config block is not synchronized across components. | ||
Most likely you will need to increase the message size limit in multiple components. | ||
### Querier | ||
Additionally, querier workers can be configured to use a larger message size limit. | ||
```yaml | ||
querier: | ||
frontend_worker: | ||
grpc_client_config: | ||
max_send_msg_size: <size> | ||
``` | ||
### Ingestion | ||
Lastly, message size is also limited in ingestion and can be modified in the distributor block. | ||
```yaml | ||
distributor: | ||
receivers: | ||
otlp: | ||
grpc: | ||
max_recv_msg_size_mib: <size> | ||
``` |