test: Yk/benchmarking scripts

benchmarking/tests/README.md

@@ -0,0 +1,95 @@
+# HOW TO USE IT
+
+This document describes how to use the benchmarking scripts.
+
+These tests are examples of benchmarking based on  Bench and HtcMock tests to measure the performances of ArmoniK.   
+We have to deploy ArmoniK on AWS cloud using the parameters files saved in folder [aws-benchmark](https://github.com/aneoconsulting/ArmoniK/tree/main/benchmarking_scripts/infrastructure/quick-deploy/aws-benchmark) to deploy ArmoniK with two partitions (bench and htcmock) with 100 pods or 1000 pods for each partition.
+
+# Deploy ArmoniK on AWS cloud  
+To deploy ArmoniK on AWS cloud with 100 pods of compute plane, from the root of the repository : 
+```bash
+cd infrastructure/quick-deploy/aws/all  
+make deploy PARAMETERS_FILE=<Path to ArmoniK>/infrastructure/quick-deploy/aws-benchmark/parameters_100pods.tfvars
+```
+To deploy ArmoniK on AWS cloud with 1000 pods of compute plane, from the root of the repository : 
+```bash
+cd infrastructure/quick-deploy/aws/all  
+make deploy PARAMETERS_FILE=<Path to ArmoniK>/infrastructure/quick-deploy/aws-benchmark/parameters_1000pods.tfvars
+```
+# Global view test scripts tree
+
+From the root of the repository, the scripts of bench and htcmock tests are in [benchmarking/tests](https://github.com/aneoconsulting/ArmoniK/tree/main/benchmarking/tests) as follows : 
+
+<pre>
+.
+├── bench
+│   ├── python-scripts
+│   │   ├── cleaner.py
+│   │   ├── merge-jsons.py
+│   │   └── reader.py
+│   ├── stats
+│   │   └── test-env.json
+│   └── test_scripts
+│       ├── bench-100k.sh
+│       ├── bench-10k.sh
+│       ├── bench-1k.sh
+│       ├── bench-5k.sh
+│       ├── htcmock-1m.sh
+│       └── test.sh
+├── htcmock
+│   ├── python-scripts
+│   │   ├── cleaner.py
+│   │   └── merge-jsons.py
+│   ├── stats
+│   │   └── test-env.json
+│   └── test-scripts
+│       ├── htcmock-1k.sh
+│       ├── htcmock-5k.sh
+│       ├── htcmock-10k.sh
+│       ├── htcmock-100k.sh
+│       ├── htcmock-1m.sh
+│       └── test.sh
+└── README.md
+</pre>
+
+# Bench & HtcMock
+## Prerequisites  
+
+Install `jsonmerge : 
+```bash
+pip install jsonmerge
+```
+## Run the tests
+* `bench-1k.sh`, `bench-5k.sh`, `bench-10k.sh`, `bench-100k.sh` and  `bench-1m.sh`: bash scripts to launch bench tests with 1000 tasks, 5000 tasks, 10000 tasks, 100000 tasks and 1000000 tasks, respectively. In addition, you can modify in the scripts other parameters like workload time, io size, ...
+* `htcmock-1k.sh`, `htcmock-5k.sh`, `htcmock-10k.sh`, `htcmock-100k.sh` and  `htcmock-1m.sh`: bash scripts to launch htcmock tests with 1000 tasks, 5000 tasks, 10000 tasks, 100000 tasks and 1000000 tasks, respectively. In addition, you can modify in the scripts other parameters like workload time, io size, ...
+* `test.sh` : script to run all the tests cases listed above and store the results in JSON files.
+
+## Clean-up the outputs
+
+* `cleaner.py` : Python script to clean the JSON output files.   
+
+* `merge_jsons.py` : Python script to merge the cleaned results files with the [versions.tfvars.json file](https://github.com/aneoconsulting/ArmoniK/tree/main/versions.tfvars.json) of the tested version of ArmoniK and `test-env.json` file (third party components of ArmoniK).
+
+
+
+## Analyze the results
+
+* `reader.py` : Python script to read the clean stats files, so we can manipulate the data of bench tests.
+
+## How to run the tests :
+
+### Linux :
+
+* Run the script `test.sh` in the directory `benchmarking/tests/bench/test-scripts` to run the tests of Bench, store the
+  outputs, clean them and merge them with the environment and infrastructure description files.
+```bash
+cd benchmarking/tests/bench/test-scripts/
+./test.sh
+```
+
+* Run the script `test.sh` in the directory `benchmarking/tests/htcmock/test-scripts` to run the tests of HtcMock, store
+  the outputs, clean them and merge them with the environment and infrastructure description files.
+```bash
+cd benchmarking/tests/htcmock/test-scripts/
+./test.sh
+```

benchmarking/tests/bench/python-scripts/cleaner.py

@@ -0,0 +1,50 @@
+#! /usr/bin/env python3
+
+import json
+import re
+import subprocess
+
+nb_pods = subprocess.run("kubectl get pod -n armonik | grep htcmock | wc -l", shell=True,
+                         stdout=subprocess.PIPE).stdout.decode('utf-8')
+nb_pods = int(nb_pods)
+
+
+def clean_file(file):
+    lines = []
+    keep_stats = []
+    keep_tput = []
+    keep_options = []
+
+    # read the output file with no needed data
+    with open(file, 'r') as f:
+        # keep only stats lines in the json file
+        lines = f.read().split("\n")
+        for line in lines:
+            if not line: continue
+            # print(line)
+            jline = json.loads(line)
+            if "stats" in jline:
+                keep_stats.append(jline["stats"])
+                keep_options.append(jline["benchOptions"])
+            if "sessionThroughput" in jline:
+                keep_tput.append(jline["sessionThroughput"])
+    dic_json = [{"Test": "bench"} | stats | options | {"throughput": tput, "nb_pods": nb_pods} for stats, tput, options
+                in
+                zip(keep_stats, keep_tput, keep_options)]
+
+    # write a clean json file with needed data
+    with open(file, 'w') as f:
+        json.dump(dic_json, f)
+
+
+# clean the stats file
+file = "../stats/1k.json"
+clean_file(file)
+file = "../stats/5k.json"
+clean_file(file)
+file = "../stats/10k.json"
+clean_file(file)
+file = "../stats/100k.json"
+clean_file(file)
+file = "../stats/1m.json"
+clean_file(file)

benchmarking/tests/bench/python-scripts/merge-jsons.py

@@ -0,0 +1,24 @@
+#! /usr/bin/env python3
+
+import json
+import re
+from jsonmerge import merge
+
+with open("../../../../versions.tfvars.json", "r") as versions:
+    infra = json.load(versions)
+with open("../stats/test-env.json", "r") as test_env:
+    env = (json.load(test_env))
+with open("../stats/1k.json", "r") as stats:
+    result_1k = json.load(stats)
+with open("../stats/5k.json", "r") as stats:
+    result_5k = json.load(stats)
+with open("../stats/10k.json", "r") as stats:
+    result_10k = json.load(stats)
+with open("../stats/100k.json", "r") as stats:
+    result_100k = json.load(stats)
+with open("../stats/1m.json", "r") as stats:
+    result_1m = json.load(stats)
+# print(merged)
+with open("../stats/results.json", "w") as r:
+    dict_json = [merge(infra, env), result_1k, result_5k, result_10k, result_100k, result_1m]
+    json.dump(dict_json, r)

benchmarking/tests/bench/python-scripts/reader.py

@@ -0,0 +1,84 @@
+#! /usr/bin/env python3
+
+import json
+from pytimeparse import parse
+import matplotlib.pyplot as plt
+import numpy as np
+
+
+# Class test_case
+# An object store a lists of the stats of all the runs of the test case
+# So we could calculate the mean and the median of each stat
+class TestCase:
+
+    def __init__(self, file):
+        self.nbtasks = []
+        self.time = []
+        self.exec_time = []
+        self.sub_time = []
+        self.retrv_time = []
+        self.throughput = []
+        self.d_parallel = []
+        self.file = file
+        with open(file) as my_bench_file:
+            data = my_bench_file.read()
+        # case = TestCase()
+        runs = json.loads(data)
+
+        for run in runs:
+            if run["nb_pods"] == 100:
+                self.nbtasks.append(run["TotalTasks"])
+                self.time.append(float(parse(run["ElapsedTime"])))
+                self.exec_time.append(float(parse(run["TasksExecutionTime"])))
+                self.sub_time.append(float(parse(run["SubmissionTime"])))
+                self.retrv_time.append(float(parse(run["ResultRetrievingTime"])))
+                self.throughput.append(float(run["throughput"]))
+                self.d_parallel.append(run["DegreeOfParallelism"])
+
+
+if __name__ == "__main__":
+
+    files = ['../stats/1k.json', '../stats/5k.json', '../stats/10k.json', '../stats/100k.json', '../stats/1m.json']
+    JsonFiles = [x for x in files if x.endswith(".json")]
+
+    # dictionary to store the stats of each test case
+    cases = {
+    }
+    cases["1k"] = TestCase(JsonFiles[0])
+    cases["5k"] = TestCase(JsonFiles[1])
+    cases["10k"] = TestCase(JsonFiles[2])
+    cases["100k"] = TestCase(JsonFiles[3])
+    cases["1m"] = TestCase(JsonFiles[4])
+
+    # Dictionary to store the mean of each test case
+    mean = {
+        "time": {},
+        "exec_time": {},
+        "sub_time": {},
+        "retrv_time": {},
+        "throughput": {}
+    }
+
+    # calculate the mean of each test case
+    for file in files:
+        filename = file.split(".")[0]
+        mean["time"][filename] = np.mean(cases[filename].time)
+        mean["exec_time"][filename] = np.mean(cases[filename].exec_time)
+        mean["sub_time"][filename] = np.mean(cases[filename].sub_time)
+        mean["retrv_time"][filename] = np.mean(cases[filename].retrv_time)
+        mean["throughput"][filename] = np.mean(cases[filename].throughput)
+
+    # print the stats
+    for file in files:
+        filename = file.split(".")[0]
+        print('Degree of parallelism of retrieving time is : ' + str(cases[filename].d_parallel[0]))
+        print('mean total time for treatment of ' + filename + ' tasks is : ' + str(
+            mean["time"][filename]) + ' s')
+        print('mean time of the execution of ' + filename + ' tasks is : ' + str(
+            mean["exec_time"][filename]) + ' s')
+        print('mean time of the submission of ' + filename + ' tasks is : ' + str(
+            mean["sub_time"][filename]) + ' s')
+        print('mean time of the retrieving of ' + filename + ' tasks is : ' + str(
+            mean["retrv_time"][filename]) + ' s')
+        print('mean throughput for ' + filename + ' tasks is : ' + str(
+            mean["throughput"][filename]) + " tasks/s \n")

benchmarking/tests/bench/stats/test-env.json

@@ -0,0 +1,16 @@
+{
+    "test_environnement" : {
+        "instance_type" : "c24.xlarge",
+        "Processor" : "Intel Xeon de 2e (Cascade Lake)",
+        "CPU_frequency" : "3.6 GHz - 3.9 GHz",
+        "nb_cpus" : "96",
+        "RAM" : "192",
+        "Network_bandwidth" : "25",
+        "EBS_bandwidth" : "19000",
+        "Kubernetes" : "AWS EKS 1.25",
+        "Object_object_type" : "AWS Elasticache for Redis",
+        "Storage_queue_storage" : "Amazon MQ, broker : ActiveMQ 5.16.4",
+        "Storage_table_type" : "MongoDB 6.0.1",
+        "OS" : "Linux"
+    }
+}

benchmarking/tests/bench/test-scripts/bench-100k.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+
+export CONTROL_PLANE_URL=$(cat ../../../infrastructure/quick-deploy/aws/all/generated/armonik-output.json | jq -r '.armonik.control_plane_url')
+
+#run  10000 tasks on 100 pods
+docker run --rm \
+	-e GrpcClient__Endpoint="${CONTROL_PLANE_URL}" \
+	-e BenchOptions__nTasks=100000 \
+	-e BenchOptions__TaskDurationMs=1 \
+	-e BenchOptions__PayloadSize=1 \
+	-e BenchOptions__ResultSize=1 \
+	-e BenchOptions__Partition=bench \
+    	-e BenchOptions__ShowEvents=false \
+	-e BenchOptions__BatchSize=50 \
+	-e BenchOptions__MaxRetries=5 \
+	-e BenchOptions__DegreeOfParallelism=5 \
+	dockerhubaneo/armonik_core_bench_test_client:$(cat ../../../../versions.tfvars.json | jq -r '.armonik_versions.core')

benchmarking/tests/bench/test-scripts/bench-10k.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+
+export CONTROL_PLANE_URL=$(cat ../../../../infrastructure/quick-deploy/aws/all/generated/armonik-output.json | jq -r '.armonik.control_plane_url')
+
+#run  10000 tasks on 100 pods
+docker run --rm \
+	-e GrpcClient__Endpoint="${CONTROL_PLANE_URL}" \
+	-e BenchOptions__nTasks=10000 \
+	-e BenchOptions__TaskDurationMs=1 \
+	-e BenchOptions__PayloadSize=1 \
+	-e BenchOptions__ResultSize=1 \
+	-e BenchOptions__Partition=bench \
+    	-e BenchOptions__ShowEvents=false \
+	-e BenchOptions__BatchSize=50 \
+	-e BenchOptions__MaxRetries=5 \
+	-e BenchOptions__DegreeOfParallelism=5 \
+	dockerhubaneo/armonik_core_bench_test_client:$(cat ../../../../versions.tfvars.json | jq -r '.armonik_versions.core')

benchmarking/tests/bench/test-scripts/bench-1k.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+
+export CONTROL_PLANE_URL=$(cat ../../../../infrastructure/quick-deploy/aws/all/generated/armonik-output.json | jq -r '.armonik.control_plane_url')
+
+#run  1000 tasks on 100 pods
+docker run --rm \
+	-e GrpcClient__Endpoint="${CONTROL_PLANE_URL}" \
+	-e BenchOptions__nTasks=1000 \
+	-e BenchOptions__TaskDurationMs=1 \
+	-e BenchOptions__PayloadSize=1 \
+	-e BenchOptions__ResultSize=1 \
+	-e BenchOptions__Partition=bench \
+    	-e BenchOptions__ShowEvents=false \
+	-e BenchOptions__BatchSize=50 \
+	-e BenchOptions__MaxRetries=5 \
+	-e BenchOptions__DegreeOfParallelism=5 \
+	dockerhubaneo/armonik_core_bench_test_client:$(cat ../../../../versions.tfvars.json | jq -r '.armonik_versions.core')

benchmarking/tests/bench/test-scripts/bench-1m.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+
+export CONTROL_PLANE_URL=$(cat ../../../../infrastructure/quick-deploy/aws/all/generated/armonik-output.json | jq -r '.armonik.control_plane_url')
+
+#run  10000 tasks on 100 pods
+docker run --rm \
+	-e GrpcClient__Endpoint="${CONTROL_PLANE_URL}" \
+	-e BenchOptions__nTasks=1000000 \
+	-e BenchOptions__TaskDurationMs=1 \
+	-e BenchOptions__PayloadSize=1 \
+	-e BenchOptions__ResultSize=1 \
+	-e BenchOptions__Partition=bench \
+    	-e BenchOptions__ShowEvents=false \
+	-e BenchOptions__BatchSize=50 \
+	-e BenchOptions__MaxRetries=5 \
+	-e BenchOptions__DegreeOfParallelism=5 \
+	dockerhubaneo/armonik_core_bench_test_client:$(cat ../../../../versions.tfvars.json | jq -r '.armonik_versions.core')

benchmarking/tests/bench/test-scripts/bench-5k.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+
+export CONTROL_PLANE_URL=$(cat ../../../../infrastructure/quick-deploy/aws/all/generated/armonik-output.json | jq -r '.armonik.control_plane_url')
+
+#run  5000 tasks on 100 pods
+docker run --rm \
+	-e GrpcClient__Endpoint="${CONTROL_PLANE_URL}" \
+	-e BenchOptions__nTasks=5000 \
+	-e BenchOptions__TaskDurationMs=1 \
+	-e BenchOptions__PayloadSize=1 \
+	-e BenchOptions__ResultSize=1 \
+	-e BenchOptions__Partition=bench \
+    	-e BenchOptions__ShowEvents=false \
+	-e BenchOptions__BatchSize=50 \
+	-e BenchOptions__MaxRetries=5 \
+	-e BenchOptions__DegreeOfParallelism=5 \
+	dockerhubaneo/armonik_core_bench_test_client:$(cat ../../../../versions.tfvars.json | jq -r '.armonik_versions.core')

benchmarking/tests/bench/test-scripts/test.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+# warm up run
+./bench-1k.sh
+
+#clearmeasured runs
+./bench-1k.sh >> ../stats/1k.json && ./bench-5k.sh >> ../stats/5k.json && ./bench-10k.sh >> ../stats/10k.json && ./bench-100k.sh >> ../stats/100k.json && ./bench-1m.sh >> ../stats/1m.json
+#clean the output file
+../python-scripts/cleaner.py
+
+#merge json files
+../python-scripts/merge-jsons.py
+
+# save and print pretty json file
+jq . ../stats/results.json > ../stats/pretty-results.json
+jq . ../stats/pretty-results.json
+
+#print the test stats and plot graphs
+#../python-scripts/reader.py
+