perf: High CPU Usage for Double-L3/L4 Proxy

[tehranian]

(See thread at: https://groups.google.com/d/msg/envoy-users/cOoLLnI0GS0/PReYVVETCQAJ)

I was doing some performance profiling of Envoy. One test case stood out: Envoy-to-Envoy L3/L4 proxy.

The setup of the tests:

• Using iperf3 to test two hosts (iperf3 client & server) in the same AZ in AWS.
• Single-Envoy: iperf-client -> Local Envoy -----> iperf-server yields ~line speed w/ 10% CPU usage on the Local Envoy instance (good).
• Double-Envoys: iperf-client -> Local Envoy -----> Remote Envoy -> iperf-server yields ~line speed w/ 10% CPU Usage on the Egress Envoy instance (good), but 40-50% CPU usage on the Ingress Envoy.

I'm able to reproduce with a configuration for the Remote Envoy as simple as this:

{
    "listeners": [{
        "name": "iperf-listener",
        "address": "tcp://0.0.0.0:15201",
        "filters": [{
            "type": "read",
            "name": "tcp_proxy",
            "config": {
                "stat_prefix": "iperf3-listener",
                "route_config": {"routes": [{"cluster": "iperf"}]}
            }
        }]
    }],
    "admin": "...",
    "cluster_manager": {
        "clusters": [{
            "name": "iperf",
            "type": "static",
            "connect_timeout_ms": 250,
            "lb_type": "round_robin",
            "hosts": [{"url": "tcp://127.0.0.1:5201"}]
        }]
    },
    "statsd_udp_ip_address": "127.0.0.1:8126"
}

My intention in that configuration is to route traffic from the public :15201 to the local :5201, which the iperf-server process is listening on.

Per suggestion in the thread, I did see CPU% drop to merely 30% on the ingress Envoy, after commenting out the following line: https://github.com/lyft/envoy/blob/cc53479c2ee01af03d549ee9c5f60bb1660f1672/source/common/filter/tcp_proxy.cc#L227

[mattklein123]

Related to #232

[mattklein123]

Going to close this given how old it is and how many changes have been made. We can revisit if there is new interest and new data.