Skip to content
This repository has been archived by the owner on May 6, 2020. It is now read-only.

Port network metrics #381

Merged
merged 8 commits into from
Aug 16, 2017
Merged

Port network metrics #381

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
edfe674
metrics: Move remaining scripts from wip
bwarden Aug 14, 2017
115b1de
metrics: add reporting to network-metrics.sh
bwarden Aug 14, 2017
05a189e
metrics: add reporting to network cpu consumption
bwarden Aug 14, 2017
2c7bb59
metrics: enable reporting for iperf3 test
bwarden Aug 14, 2017
c6baf9f
metrics: add reporting for nuttcp test
bwarden Aug 14, 2017
d2b08e3
metrics: add reporting to nginx test
bwarden Aug 14, 2017
f67f475
metrics: Add reporting to network memory tests
bwarden Aug 14, 2017
b49d7e9
metrics: Add network metrics to run_all_metrics
bwarden Aug 14, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions ...rk/wip/network-metrics-cpu-consumption.sh → ...etwork/network-metrics-cpu-consumption.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,11 +64,13 @@ function cpu_consumption {
echo >&2 "WARNING: sleeping for $middle_time seconds in order to have server and client stable"
sleep ${middle_time}
qemu_pids=$(pidof ${QEMU_PATH})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, a note/thought. Probably not going to block this for the minute, but...
I ran the tests initially with RUNTIME=runc, because my local CC install may have been a little in-flux... and of course runc does not have any QEMU processes, so we actually bomb or fail some of the tests (like when we try to do a ps of the pids of QEMU, we fail the ps, and then we get a divide by zero error etc.

So, later maybe, we should make the scripts spot that there is no QEMU, and gracefully-ish fail.
That, or we make run_all_metrics.sh more intelligent about which tests can be run for which runtimes, maybe.

But, for now, we'd really like to be running these tests with CC3.x, so I will install and test that...

ps --no-headers -o %cpu -p "$qemu_pids" > "$result"
sed -i 's/ //g' "$result"
local total_cpu_consumption=$(awk '{ total += $1 } END { print total/NR }' "$result")
local total_cpu_consumption=$(ps --no-headers -o %cpu \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

much nicer without the temp file, thanks!

-p $(pidof ${QEMU_PATH}) | awk '{ total+= $1 } END { print total/NR }')
echo "The cpu % consumption is : $total_cpu_consumption"

save_results "network metrics cpu consumption" "" \
"$total_cpu_consumption" "%"

clean_environment "$server_name"
$DOCKER_EXE rm -f ${client_name} > /dev/null
}
Expand Down
54 changes: 46 additions & 8 deletions ...ics/network/wip/network-metrics-iperf3.sh → metrics/network/network-metrics-iperf3.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ extra_args="-ti --rm"

set -e

## Test name for reporting purposes
test_name="network metrics iperf3"

# This script will perform all the measurements using a local setup using iperf3

# Test single direction TCP bandwith
Expand All @@ -47,8 +50,16 @@ function iperf3_bandwidth {
local client_command="mount -t ramfs -o size=20M ramfs /tmp && iperf3 -c ${server_address} -t ${time}"
start_client "$extra_args" "$client_name" "$image" "$client_command" > "$result"

local total_bandwidth=$(cat $result | tail -n 3 | head -1 | awk '{print $(NF-2), $(NF-1)}')
echo "Network bandwidth is : $total_bandwidth"
local result_line=$(grep -m1 -E '\breceiver\b' $result)
local -a results
read -a results <<< $result_line
local total_bandwidth=${results[6]}
local total_bandwidth_units=${results[7]}
echo "Network bandwidth is : $total_bandwidth $total_bandwidth_units"

save_results "${test_name}" "network bandwidth" \
"$total_bandwidth" "$total_bandwidth_units"

clean_environment "$server_name"
}

Expand All @@ -61,8 +72,16 @@ function iperf3_jitter {
local client_command="mount -t ramfs -o size=20M ramfs /tmp && iperf3 -c ${server_address} -u -t ${time}"
start_client "$extra_args" "$client_name" "$image" "$client_command" > "$result"

local total_jitter=$(cat $result | tail -n 4 | head -1 | awk '{print $(NF-4), $(NF-3)}')
echo "Network jitter is : $total_jitter"
local result_line=$(grep -m1 -A1 -E '\bJitter\b' $result | tail -1)
local -a results
read -a results <<< $result_line
local total_jitter=${results[8]}
local total_jitter_units=${results[9]}
echo "Network jitter is : $total_jitter $total_jitter_units"

save_results "${test_name}" "network jitter" \
"$total_jitter" "$total_jitter_units"

clean_environment "$server_name"
}

Expand All @@ -75,10 +94,29 @@ function iperf3_bidirectional_bandwidth_client_server {
local client_command="mount -t ramfs -o size=20M ramfs /tmp && iperf3 -c ${server_address} -d -t ${time}"
start_client "$extra_args" "$client_name" "$image" "$client_command" > "$result"

local total_bidirectional_client_bandwidth=$(cat $result | tail -n 3 | head -1 | awk '{print $(NF-2), $(NF-1)}')
local total_bidirectional_server_bandwidth=$(cat $result | tail -n 4 | head -1 | awk '{print $(NF-3), $(NF-2)}')
echo "Network bidirectional bandwidth (client to server) is : $total_bidirectional_client_bandwidth"
echo "Network bidirectional bandwidth (server to client) is : $total_bidirectional_server_bandwidth"
local client_result=$(grep -m1 -E '\breceiver\b' $result)
local server_result=$(grep -m1 -E '\bsender\b' $result)
local -a client_results
read -a client_results <<< ${client_result}
read -a server_results <<< ${server_result}
local total_bidirectional_client_bandwidth=${client_results[6]}
local total_bidirectional_client_bandwidth_units=${client_results[7]}
local total_bidirectional_server_bandwidth=${server_results[6]}
local total_bidirectional_server_bandwidth_units=${server_results[7]}
echo "Network bidirectional bandwidth (client to server) is :" \
"$total_bidirectional_client_bandwidth" \
"$total_bidirectional_client_bandwidth_units"
echo "Network bidirectional bandwidth (server to client) is :" \
"$total_bidirectional_server_bandwidth" \
"$total_bidirectional_server_bandwidth_units"

save_results "${test_name}" "network bidir bw client to server" \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, and here I see what you did with the multi-save of results into a single file... I think that is OK, but you are 'pushing the boundaries' here of what we have in place, so, some notes:

  • use of the Args field is OK. That was not its original intention (it was to store the args the test had used...), but, it is not really being used or useful for that right now - so, I'd call that acceptable...
  • placing multiple mixed results in a single results file will almost definitely break https://github.com/clearcontainers/tests/tree/master/cmd/checkmetrics parsing of those files... I think we can do an interim fix (filter the CSV files with grep for instance right now to get the results we need to check), and then add Args filtering to checkmetrics.

So, not the end of the world, and a pragmatic step forwards in holding mixed and varied results.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it makes more sense to put each result in an independent file, that's fine, and pretty easy to accomplish. I'd just take the unique info out of Args and append it to Test Name. Let me know if you'd prefer that, or adding another field, etc...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've mulled over this a bit. I'm going to make the call to keep the data together for the minute in a single file. Now that you extract more of the results from some of the tests, I think if we put those into separate files then there is some risk in the future that we'd end up with a set of files and no proof that the data came from the same test run (well, OK, maybe timestamps and commit IDs could give a heavy clue... still...). So, let's leave as is, and then we will fix up checkmetrics and related other scripts to pull out the data they are looking for.

"$total_bidirectional_client_bandwidth" \
"$total_bidirectional_client_bandwidth_units"
save_results "${test_name}" "network bidir bw server to client" \
"$total_bidirectional_server_bandwidth" \
"$total_bidirectional_server_bandwidth_units"

clean_environment "$server_name"
}

Expand Down
7 changes: 5 additions & 2 deletions ...work/wip/network-metrics-memory-pss-1g.sh → .../network/network-metrics-memory-pss-1g.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,14 @@ function pss_memory {
echo >&2 "WARNING: sleeping for $middle_time seconds in order to sample the PSS"
sleep ${middle_time}

local memory_command="smem -c pss"
${memory_command} -P ${QEMU_PATH} | tail -n 2 > "$result"
local memory_command="smem --no-header -c pss"
${memory_command} -P ${QEMU_PATH} > "$result"

local memory=$(awk '{ total += $1 } END { print total/NR }' "$result")
echo "The PSS memory is : $memory Kb"

save_results "network metrics memory" "pss 1g" "$memory" "kb"

echo >&2 "WARNING: This test is being affected by https://github.com/01org/cc-oci-runtime/issues/795"
clean_environment "$server_name"
$DOCKER_EXE rm -f ${client_name} > /dev/null
Expand Down
6 changes: 5 additions & 1 deletion ...network/wip/network-metrics-memory-pss.sh → ...ics/network/network-metrics-memory-pss.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,10 +63,14 @@ function pss_memory {
# Measurement after client and server are more stable
echo >&2 "WARNING: sleeping for $middle_time seconds in order to have server and client stable"
sleep ${middle_time}
smem -c "pss" -P ${QEMU_PATH} | tail -n 2 > "$result"
local memory_command="smem --no-header -c pss"
${memory_command} -P ${QEMU_PATH} > "$result"

local total_pss_memory=$(awk '{ total += $1 } END { print total/NR }' "$result")
echo "The PSS memory is : $total_pss_memory Kb"

save_results "network metrics memory" "pss" "$total_pss_memory" "kb"

clean_environment "$server_name"
$DOCKER_EXE rm -f ${client_name} > /dev/null
}
Expand Down
7 changes: 5 additions & 2 deletions ...work/wip/network-metrics-memory-rss-1g.sh → .../network/network-metrics-memory-rss-1g.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,10 +65,13 @@ function rss_memory {
echo >&2 "WARNING: sleeping for $middle_time seconds in order to sample the RSS"
sleep ${middle_time}

local memory_command="smem -c rss"
${memory_command} -P ${QEMU_PATH} | tail -n 2 > "$result"
local memory_command="smem --no-header -c rss"
${memory_command} -P ${QEMU_PATH} > "$result"

local memory=$(awk '{ total += $1 } END { print total/NR }' "$result")
echo "The RSS memory is : $memory Kb"

save_results "network metrics memory" "rss 1g" "$memory" "kb"

echo >&2 "WARNING: This test is being affected by https://github.com/01org/cc-oci-runtime/issues/795"
clean_environment "$server_name"
Expand Down
27 changes: 23 additions & 4 deletions ...ics/network/wip/network-metrics-nuttcp.sh → metrics/network/network-metrics-nuttcp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@ set -e

# Test single docker->docker udp bandwidth

## Test name for reporting purposes
test_name="network metrics nuttcp"

function udp_bandwidth {
# Currently default nuttcp has a bug
# see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=745051
Expand All @@ -52,10 +55,26 @@ function udp_bandwidth {
$DOCKER_EXE exec ${server_name} sh -c "${server_command}"
start_client "$extra_args" "$client_name" "$image" "$client_command" > "$result"

local total_bandwidth=$(cat "$result" | tail -1 | cut -d'=' -f2 | awk '{print $(NF-11), $(NF-10)}')
local total_loss=$(cat "$result" | tail -1 | awk '{print $(NF-1), $(NF)}')
echo "UDP bandwidth is (${bl} buffer size) : $total_bandwidth"
echo "UDP % of packet loss is (${bl} buffer size) : $total_loss"
# 6, 7, 16, "%"

local result_line=$(tail -1 ${result})
local -a results
read -a results <<< ${result_line%$'\r'}
local total_bandwidth=${results[6]}
local total_bandwidth_units=${results[7]}
local total_loss=${results[16]}
local total_loss_units="%"
echo "UDP bandwidth is (${bl} buffer size) :" \
"$total_bandwidth $total_bandwidth_units"
echo "UDP % of packet loss is (${bl} buffer size) :" \
"${total_loss}${total_loss_units}"

local subtest_name="UDP ${bl}b buffer"
save_results "${test_name}" "${subtest_name} bandwidth" \
"$total_bandwidth" "$total_bandwidth_units"
save_results "${test_name}" "${subtest_name} packet loss" \
"$total_loss" "$total_loss_units"

clean_environment "$server_name"
}

Expand Down
27 changes: 23 additions & 4 deletions metrics/network/wip/network-metrics.sh → metrics/network/network-metrics.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,10 +49,29 @@ function bidirectional_bandwidth_server_client {
client_command="iperf -c ${server_address} -d -t ${time}"
start_client "$extra_args" "$client_name" "$image" "$client_command" > "$result"

local total_bidirectional_server_bandwidth=$(cat $result | tail -1 | awk '{print $(NF-1), $NF}')
local total_bidirectional_client_bandwidth=$(cat $result | tail -n 2 | head -1 | awk '{print $(NF-1), $NF}')
echo "Bi-directional network bandwidth is (client to server) : $total_bidirectional_client_bandwidth"
echo "Bi-directional network bandwidth is (server to server) : $total_bidirectional_server_bandwidth"
local server_result=$(tail -1 $result)
read -a server_results <<< ${server_result%$'\r'}
local client_result=$(tail -n 2 $result | head -1)
read -a client_results <<< ${client_result%$'\r'}

local total_bidirectional_server_bandwidth=${server_results[-2]}
local total_bidirectional_server_bandwidth_units=${server_results[-1]}
local total_bidirectional_client_bandwidth=${client_results[-2]}
local total_bidirectional_client_bandwidth_units=${client_results[-1]}
echo "Bi-directional network bandwidth is (client to server) :" \
"$total_bidirectional_client_bandwidth" \
"$total_bidirectional_client_bandwidth_units"
echo "Bi-directional network bandwidth is (server to client) :" \
"$total_bidirectional_server_bandwidth" \
"$total_bidirectional_server_bandwidth_units"

save_results "network metrics" "client to server" \
"$total_bidirectional_client_bandwidth" \
"$total_bidirectional_client_bandwidth_units"
save_results "network metrics" "server to client" \
"$total_bidirectional_server_bandwidth" \
"$total_bidirectional_server_bandwidth_units"

clean_environment "$server_name"
}

Expand Down
10 changes: 8 additions & 2 deletions ...network/wip/network-nginx-ab-benchmark.sh → ...ics/network/network-nginx-ab-benchmark.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,10 +69,16 @@ function nginx_ab_networking {
sleep "$sleep_secs"
ab -n ${requests} -c ${concurrency} http://${url}/ > "$total_requests"

result=$(cat "$total_requests" | grep "Requests per second" | cut -d ':' -f2 | sed 's/ //g' | cut -d '[' -f1)
cp -p $total_requests result.test

local result=$(grep "^Requests per second" $total_requests)
[[ $result =~ :[[:blank:]]*([[:digit:]]+(\.[[:digit:]]*)?) ]] && rps=${BASH_REMATCH[1]}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Argh, my eyes! ;-) Well, OK, it is more efficient than the pipelined version, and in some ways more understandable as it pretty much lays out the format that it is parsing...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I avoided leaning toothpicks :) I considered pasting sample data in a comment above, but realized I'd probably have to sanitize that since it's actual benchmarking results.

$DOCKER_EXE rm -f ${container_name} > /dev/null
rm -f "$total_requests"
}

nginx_ab_networking
echo "The total of requests per second is : $result"
echo "The total of requests per second is : $rps"
save_results "network nginx ab benchmark" \
"requests=${requests} concurrency=${concurrency}" \
"$rps" "requests/s"
8 changes: 8 additions & 0 deletions metrics/run_all_metrics.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,11 @@ bash ${SCRIPT_PATH}/time/docker_workload_time.sh true busybox $RUNTIME 100

# Run the network metrics
bash ${SCRIPT_PATH}/network/network-latency.sh
bash ${SCRIPT_PATH}/network/network-metrics-cpu-consumption.sh
bash ${SCRIPT_PATH}/network/network-metrics-iperf3.sh
bash ${SCRIPT_PATH}/network/network-metrics-memory-pss-1g.sh
bash ${SCRIPT_PATH}/network/network-metrics-memory-pss.sh
bash ${SCRIPT_PATH}/network/network-metrics-memory-rss-1g.sh
bash ${SCRIPT_PATH}/network/network-metrics-nuttcp.sh
bash ${SCRIPT_PATH}/network/network-metrics.sh
bash ${SCRIPT_PATH}/network/network-nginx-ab-benchmark.sh