Only ~100 kbps throughput in secnetperf test

[furrio]

I want to evaluate the maximum throughput of peer-to-peer transmission. The NIC I'm using is from Mellanox CX-6 series. The port rate is 100Gbps, but the throughput of quic is only around 100kps, for both local and peer-to-peer testing.

I run the local test as follow:

In section 1:

PS /home/ceni-a1/msquic/artifacts/bin/linux/x64_Debug_openssl> ./secnetperf -exec:maxtput
Started!

In section 2:

PS /home/ceni-a1/msquic/artifacts/bin/linux/x64_Debug_openssl> ./secnetperf -target:localhost -exec:maxtput -down:5s -ptput:1
Started!

Result: Download 131072 bytes @ 104 kbps (10005.009 ms).
App Main returning status 0

I monitored the CPU usage during the test and the usage is almost zero. In the same scenario, TCP throughput can reach 13G and CPU usage can reach 100%.

I build the repository on Ubuntu18.04.

[nibanks]

Can you try a release build?

[furrio]

Changing to the release build improves performance for a little bit, but still well below the the benchmark.

PS /home/ceni-a1/msquic/artifacts/bin/linux/x64_Release_openssl> ./secnetperf -target:localhost -exec:maxtput -down:10s -ptput:1
Started!

Result: Download 24817913 bytes @ 19825 kbps (10014.540 ms).
App Main returning status 0
PS /home/ceni-a1/msquic/artifacts/bin/linux/x64_Release_openssl> ./secnetperf -target:localhost -exec:maxtput -up:10mb -streams:10 -ptput:1
Started!

Result: Upload 10000000 bytes @ 8408 kbps (9514.238 ms).
Result: Upload 10000000 bytes @ 6183 kbps (12938.695 ms).
Result: Upload 10000000 bytes @ 4805 kbps (16645.940 ms).
Result: Upload 10000000 bytes @ 3945 kbps (20277.119 ms).
Result: Upload 10000000 bytes @ 3163 kbps (25291.560 ms).
Result: Upload 10000000 bytes @ 2989 kbps (26758.182 ms).
Result: Upload 10000000 bytes @ 2948 kbps (27132.617 ms).
Result: Upload 10000000 bytes @ 2457 kbps (32548.429 ms).
Result: Upload 10000000 bytes @ 2308 kbps (34655.189 ms).
Result: Upload 10000000 bytes @ 2232 kbps (35831.396 ms).
App Main returning status 0

[nibanks]

Try setting -pconn:1 -pstream:1 on both sides. It will print some statistics that helps diagnose performance issues.

[furrio]

PS /home/ceni-a1/msquic/artifacts/bin/linux/x64_Release_openssl> ./secnetperf -target:localhost -exec:maxtput -down:10s -ptput:1 -pconn:1 -pstream:1
Started!

Stream Timings (flow blocked):
  SCHEDULING:               0 us
  PACING:                   2809 us
  AMPLIFICATION_PROT:       0 us
  CONGESTION_CONTROL:       234 us
  CONN_FLOW_CONTROL:        0 us
  STREAM_ID_FLOW_CONTROL:   0 us
  STREAM_FLOW_CONTROL:      0 us
  APP:                      0 us
Result: Download 24765933 bytes @ 19778 kbps (10017.328 ms).
Connection Statistics:
  RTT                       218 us
  MinRTT                    88 us
  EcnCapable                0
  SendTotalPackets          632
  SendSuspectedLostPackets  0
  SendSpuriousLostPackets   0
  SendCongestionCount       0
  SendEcnCongestionCount    0
  RecvTotalPackets          17211
  RecvReorderedPackets      0
  RecvDroppedPackets        0
  RecvDuplicatePackets      0
  RecvDecryptionFailures    0
App Main returning status 0

Disable pacing:

PS /home/ceni-a1/msquic/artifacts/bin/linux/x64_Release_openssl> ./secnetperf -target:localhost -exec:maxtput -down:10s -ptput:1 -pconn:1 -pstream:1 -pacing:0
Started!

Stream Timings (flow blocked):
  SCHEDULING:               0 us
  PACING:                   0 us
  AMPLIFICATION_PROT:       0 us
  CONGESTION_CONTROL:       493 us
  CONN_FLOW_CONTROL:        0 us
  STREAM_ID_FLOW_CONTROL:   0 us
  STREAM_FLOW_CONTROL:      0 us
  APP:                      0 us
Result: Download 25039479 bytes @ 20002 kbps (10014.437 ms).
Connection Statistics:
  RTT                       202 us
  MinRTT                    137 us
  EcnCapable                0
  SendTotalPackets          645
  SendSuspectedLostPackets  0
  SendSpuriousLostPackets   0
  SendCongestionCount       0
  SendEcnCongestionCount    0
  RecvTotalPackets          17407
  RecvReorderedPackets      0
  RecvDroppedPackets        0
  RecvDuplicatePackets      0
  RecvDecryptionFailures    0
App Main returning status 0

[nibanks]

Can you do an upload and share the numbers? I need the output on the sender side.

[furrio]

Looks like congestion control took a lot of time.

PS /home/ceni-a1/msquic/artifacts/bin/linux/x64_Release_openssl> ./secnetperf -target:localhost -exec:maxtput -up:5s -ptput:1 -pconn:1 -pstream:1 -cc:cubic
Started!

Stream Timings (flow blocked):
  SCHEDULING:               16 us
  PACING:                   2116 us
  AMPLIFICATION_PROT:       0 us
  CONGESTION_CONTROL:       4962747 us
  CONN_FLOW_CONTROL:        0 us
  STREAM_ID_FLOW_CONTROL:   0 us
  STREAM_FLOW_CONTROL:      0 us
  APP:                      55179 us
Result: Upload 13893632 bytes @ 22006 kbps (5050.752 ms).
Connection Statistics:
  RTT                       309 us
  MinRTT                    87 us
  EcnCapable                0
  SendTotalPackets          10201
  SendSuspectedLostPackets  495
  SendSpuriousLostPackets   0
  SendCongestionCount       11
  SendEcnCongestionCount    0
  RecvTotalPackets          424
  RecvReorderedPackets      0
  RecvDroppedPackets        0
  RecvDuplicatePackets      0
  RecvDecryptionFailures    0
App Main returning status 0
PS /home/ceni-a1/msquic/artifacts/bin/linux/x64_Release_openssl> ./secnetperf -target:localhost -exec:maxtput -up:5s -ptput:1 -pconn:1 -pstream:1 -cc:bbr
Started!

Stream Timings (flow blocked):
  SCHEDULING:               181 us
  PACING:                   0 us
  AMPLIFICATION_PROT:       0 us
  CONGESTION_CONTROL:       4938207 us
  CONN_FLOW_CONTROL:        0 us
  STREAM_ID_FLOW_CONTROL:   0 us
  STREAM_FLOW_CONTROL:      0 us
  APP:                      434412 us
Result: Upload 13500416 bytes @ 21326 kbps (5064.329 ms).
Connection Statistics:
  RTT                       230 us
  MinRTT                    78 us
  EcnCapable                0
  SendTotalPackets          20993
  SendSuspectedLostPackets  11520
  SendSpuriousLostPackets   0
  SendCongestionCount       196
  SendEcnCongestionCount    0
  RecvTotalPackets          514
  RecvReorderedPackets      0
  RecvDroppedPackets        0
  RecvDuplicatePackets      0
  RecvDecryptionFailures    0
App Main returning status 0