Flow Aggregator performance benchmarking #2188
Conversation
Used pprof to see the bottlenecks in cpuprofile and memprofile. From above result, fixed |
|
@antoninbas Added one test, where we benchmark flow aggregator with unique Intra-Node flow records from 10 exporters. Currently, I added 1000 records per exporter. |
Codecov Report
@@ Coverage Diff @@
## main #2188 +/- ##
==========================================
- Coverage 60.66% 60.66% -0.01%
==========================================
Files 285 285
Lines 23032 23033 +1
==========================================
- Hits 13973 13972 -1
- Misses 7553 7554 +1
- Partials 1506 1507 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
|
|
I haven't done any real review yet, but my first comment would be about whether we could make this more "realistic" by having 1,000 exporters (which would correspond to 1,000 Antrea Agents), and have them export 100 records per second each. BTW, I am not sure that the Go benchmark framework is the best suited here. I feel like we want to be evaluate the performance of the Flow Aggregator in a more isolated manner, without the impact of the test exporters / collector. What do you think? |
Yeah, a higher number of exporters and 100 records per second sound realistic. I am currently doing 50 records per 0.25 seconds.
Yes, I thought about that. Currently, we run the benchmark for the whole application. We could do for a single goroutine using pprof labels. What do you think of this approach? |
Maybe, but doesn't it constrain us to a single goroutine? The aggregator starts multiple goroutines IIRC (for the collecting process, aggregating process, etc.) and in the future, we may even want to be support adding extra goroutines for the same functionality to scale horizontally. I think if you want to benchmark a specific "part" of the aggregator (e.g. collecting process), then the gotest benchmarking capabilities and the current approach overall make a lot of sense. But if we want to "scale test" the Flow Aggregator, running is as its own process and measuring resource usage this way may be better. This is also what we did for the Controller I think. We have some benchmarks for the NetworkPolicy Controller (https://github.com/antrea-io/antrea/blob/main/pkg/controller/networkpolicy/networkpolicy_controller_perf_test.go) and we ran some scale testing using the agent simulator to simulate a lot of connections and consummers. |
Yes, we do have multiple goroutines in Flow Aggregator. I read that we could propagate labels/context from parent goroutine. I gave it a shot but found that applying pprof labels is not straightforward in the benchmark test without changing the application code.
Got it. With the current benchmarking and profiling methodology, we found some more bottlenecks in go-ipfix code. We are working on fixing those issues. |
|
Made some fixes in go-ipfix to improve the performance: vmware/go-ipfix#204 Before go-ipfix fix: After go-ipfix fix: 21.5% reduction in memory usage and 17.5% reduction in allocs. [EDIT] 70% reduction in memory usage, 36% reduction in allocations, and 3% increase in messages processed and received by external collector. |
zyiou
added
antrea/flow-visibility/test
area/flow-visibility
srikartati
force-pushed
the
perf_flow_agg
branch
6 times, most recently
from
a8252a6
to
069d93b
Compare
|
/test-ipv6-e2e |
srikartati
force-pushed
the
perf_flow_agg
branch
3 times, most recently
from
4b8e203
to
a2145d7
Compare
Added test for IntraFlowRecords. Simulated 10 exporters and each of them send 1000 flow records. Used benchmark in-built tools like cpuprofile, memprofile, benchmem etc., to get the performance metrics. Signed-off-by: Srikar Tati <[email protected]>
srikartati
force-pushed
the
perf_flow_agg
branch
from
82f2182
to
2720db5
Compare
Added test for IntraFlowRecords.
Simulated 10 exporters and each of them send 1000 flow records.
Used benchmark in-built tools like cpuprofile, memprofile, benchmem, etc.,
to get the performance metrics.