Micro benchmarks

[fpetrogalli]

Use the google benchmark library [1] to write micro-benchmarks for the vector math routines.

Each benchmark should be invoking the function with random values in the input range.

[1] https://github.com/google/benchmark

[shibatch]

I am now considering to overhaul the benchmarking tools.
My plan is to automate graphing with gnuplot first, and then introduce Google's framework.
In order to process data, I am planning to use java, since I am used to it.

[fpetrogalli]

I think we should do this after the transition from Makefile to cmake, as all the components will be easier to integrate.

I think that it would be better to implement this in the following order:

1. write the micro benchmarks with the google benchmark framework;
2. produce the graphic visualization.

I am happy for you to use Java+gnuplot, but I think that it would be better to use python+mathplotlib, as it might be easier to maintain and run. It will require less configuration than setting up a Java VM, and I think it will be easier to deploy on Travis.

[shibatch]

Do we need to deploy it on Travis?
You know, the benchmark results cannot be trusted since Travis is on a cloud.
The problem with python is that I need time to learn those things.
I have lots of experience with Java, and some experience with gnuplot, so it would be much easier for me.

[fpetrogalli]

Then, Java + gnuplot is, you are right, we cannot run benchmarks on the cloud.

I still think you should first use the micro benchmark system of google and then plot the data, because the output of Google benchmark might be different from what you expect in the plotting scripts. You might need to rework the scripts if that is going to be the case. Also, Google benchmark has some report facilities (see [1] for example), which might be used instead of the plotting tool.

I recommend you to leave the scripts for plotting as the last (optional) part of this work.

[1] http://1.bp.blogspot.com/-wk7hsdYodo8/UtS75FZag6I/AAAAAAAAAks/cQFiCXPbtwk/s1600/image00.png

[shibatch]

Could you explain a little bit about how you are planning to use the output data?
I have been always thought this work as a part of writing our paper.
From that perspective, drawing graph is essential.

[fpetrogalli]

Getting numbers is very important. Using google framework should guarantee that we have reliable evaluations. If the aim of setting up this is only to get the numbers in the paper, I think we don't need to store any script in here for that.

My goal here was to make sure that the numbers we were getting where reliable, and I believe that those we could get with google micro benchmarks framework have such property. They would be reliable not only for us, but also for people reading these numbers are they are produce via a standard tool.

Also, using google micro benchmarks would make easier for other people to verify our claims on their own machines.

For the paper, maybe we could store the scripts that generate the graphics in the paper itself?

[shibatch]

I still don't understand why there could be so much difference in reliability of measured values. We are just measuring execution time of small C codes, which basically don't have conditional branch or memory access. Execution time is highly reproducible. If this were java, there would be many things to consider like JIT or garbage collection.
How much reliability do you need?

[joanaxcruz]

Reviving discussion in this issue, as new changes are predicted to come in place in order to close it.

Above it's discussed introducing googlebench framework. I decided to investigate a bit on this, and the following points made it appealing to the project:

• active repo - https://github.com/google/benchmark
• widely used in other projects
• reduced maintenance cost - this would fall on the people maintaining the framework
• straightforward integration with CMake
• easy to use
• comes with a lot of free functionality with it (like filtering the benchmarks to only show performance of a particular function, and past the results in json or csv format).
• reliable results (results don't vary between runs)

In order to close this issue I propose the following plan:

1. Create a new benchmark tool using googlebench (making sure all functionality from current tool is migrated to new tool)
2. Treat the results and display results using python and plotly (interactive graphs)

Future work not necessary to close this issue:

• Create a GHA benchmarking pipeline

[blapie]

Looking very good! Looking forward to seeing new results.

• Create a GHA benchmarking pipeline

I don't know how useful it would be to run in GHA, these are shared resources so results might be polluted. Last time I googled it, it seemed that benchmarks in GHA were not that common. But maybe in practice figures can still be useful.
We should definitely at least try to build benchmarks in pre/post-commit though.