Increase default max size and add troubleshotting page

CHANGELOG.md

@@ -1,6 +1,7 @@
 ## main / unreleased
 
 * [CHANGE] tempo: check configuration returns now a list of warnings [#1663](https://github.com/grafana/tempo/pull/1663) (@frzifus)
+* [CHANGE] Increase default values for `server.grpc_server_max_recv_msg_size` and `server.grpc_server_max_send_msg_size` from 4MB to 16MB [#1688](https://github.com/grafana/tempo/pull/1688) (@mapno)
 * [ENHANCEMENT] metrics-generator: expose span size as a metric [#1662](https://github.com/grafana/tempo/pull/1662) (@ie-pham)
 * [ENHANCEMENT] Set Max Idle connections to 100 for Azure, should reduce DNS errors in Azure [#1632](https://github.com/grafana/tempo/pull/1632) (@electron0zero)
 

cmd/tempo/app/config.go

@@ -71,6 +71,10 @@ func (c *Config) RegisterFlagsAndApplyDefaults(prefix string, f *flag.FlagSet) {
 	flagext.DefaultValues(&c.Server)
 	c.Server.LogLevel.RegisterFlags(f)
 
+	// Increase max message size to 16MB
+	c.Server.GPRCServerMaxRecvMsgSize = 16 * 1024 * 1024
+	c.Server.GRPCServerMaxSendMsgSize = 16 * 1024 * 1024
+
 	// The following GRPC server settings are added to address this issue - https://github.com/grafana/tempo/issues/493
 	// The settings prevent the grpc server from sending a GOAWAY message if a client sends heartbeat messages
 	// too frequently (due to lack of real traffic).

docs/tempo/website/troubleshooting/_index.md

@@ -21,3 +21,4 @@ In addition, the [Tempo runbook](https://github.com/grafana/tempo/blob/main/oper
 - [Error message "Too many jobs in the queue"]({{< relref "too-many-jobs-in-queue/" >}})
 - [Queries fail with 500 and "error using pageFinder"]({{< relref "bad-blocks/" >}})
 - [I can search traces, but there are no service name or span name values available]({{< relref "search-tag" >}})
+- [Error message "response larger than the max (<number> vs <limit>)]({{< relref "response-too-large/" >}})

docs/tempo/website/troubleshooting/response-too-large.md

@@ -0,0 +1,55 @@
+---
+title: Response larger than the max
+weight: 477
+---
+
+# Response too large
+
+The error message will take a similar form to the following:
+
+```
+500 Internal Server Error Body: response larger than the max (<size> vs <limit>)
+```
+
+This error indicates that the response received or sent is too large.
+This can happen in multiple places, but it's most commonly seen in the query path,
+with messages between the querier and the query frontend.
+
+## Solutions
+
+### Tempo server (general)
+
+Tempo components communicate with each other via gRPC requests.
+To increase the maximum message size, you can increase the gRPC message size limit in the server block.
+
+```yaml
+server:
+  grpc_server_max_recv_msg_size: <size>
+  grpc_server_max_send_msg_size: <size>
+```
+
+The server config block is not synchronized across components.
+Most likely you will need to increase the message size limit in multiple components.
+
+### Querier
+
+Additionally, querier workers can be configured to use a larger message size limit.
+
+```yaml
+querier:
+    frontend_worker:
+        grpc_client_config:
+            max_send_msg_size: <size>
+```
+
+### Ingestion
+
+Lastly, message size is also limited in ingestion and can be modified in the distributor block.
+
+```yaml
+distributor:
+  receivers:
+    otlp:
+      grpc:
+        max_recv_msg_size_mib: <size>
+```