Add perf exporter

[hodgesds]

This implements #1238 by adding perf based profiling metrics. It's still a work in progress but so far it seems to be working locally. I'd like to instrument more of the already available metrics but figured I'd put this out there for others to see.

[hodgesds]

Example of L1 data cache hit rate:

[SuperQ]

I'm going to have to dig deeper into the new included libraries, but it looks like so far everything is being returned as gauge values.

We prefer to expose raw underlying counters in Prometheus, rather than try to use pre-calculated rates. Is this possible with perf?

[hodgesds]

Yeah, most of the underlying metrics are counters and I just copy pasted some of the other exporter code to get this started. I think most of the metrics will be mapped to counters. The one thing I need to figure out is how are overflows handled normally for other exporters.

[SuperQ]

For counters, they should all be named _total and use prometheus.CounterValue.

If you have concerns about 2^53 uint64 overflows, take a look at how we handle it in the snmp_exporter.

[hodgesds]

Pushed up changes, I think this should be good if you want to test it locally.

[SuperQ]

I started with echo 2 > sudo tee /proc/sys/kernel/perf_event_paranoid.

Then I ran a couple collections and it seems to be resetting counters after each scrape:

$ curl -s 'http://localhost:9100/metrics?collect\[\]=perf' | grep node_perf_page_faults_total
# HELP node_perf_page_faults_total Number of page faults
# TYPE node_perf_page_faults_total counter
node_perf_page_faults_total{cpu="0"} 16
node_perf_page_faults_total{cpu="1"} 0
node_perf_page_faults_total{cpu="2"} 60
node_perf_page_faults_total{cpu="3"} 1796
node_perf_page_faults_total{cpu="4"} 31
node_perf_page_faults_total{cpu="5"} 0
node_perf_page_faults_total{cpu="6"} 42
node_perf_page_faults_total{cpu="7"} 5
$ curl -s 'http://localhost:9100/metrics?collect\[\]=perf' | grep node_perf_page_faults_total
# HELP node_perf_page_faults_total Number of page faults
# TYPE node_perf_page_faults_total counter
node_perf_page_faults_total{cpu="0"} 2
node_perf_page_faults_total{cpu="1"} 3
node_perf_page_faults_total{cpu="2"} 10
node_perf_page_faults_total{cpu="3"} 0
node_perf_page_faults_total{cpu="4"} 0
node_perf_page_faults_total{cpu="5"} 10
node_perf_page_faults_total{cpu="6"} 4
node_perf_page_faults_total{cpu="7"} 62

Sadly, this seems to persist across restarts of the exporter as well.

[hodgesds]

I'm not sure what capabilities you are running so that may play a factor. The one thing to note from the perf_event_open man page:

pid == -1 and cpu >= 0
This measures all processes/threads on the specified CPU. This requires CAP_SYS_ADMIN capability or a /proc/sys/kernel/perf_event_paranoid value of less than 1.

Due to the fact that it is currently configured to trace all processes on the specific CPU I doubt a paranoid value of 2 (allow only user-space measurements) would work.

[SuperQ]

Even with -1, I still get reset counters.

I'm also seeing this error after a few scrapes:

2019/03/05 17:40:03 http: Accept error: accept tcp [::]:9100: accept4: too many open files; retrying in 5ms

Looks like there may be a leak.

[hodgesds]

Real dumb mistake, I was assuming that NewPerfCollector was only being called once and that certainly is not the case. I was able to replicate your results and changed the code so that the profilers are wrapped in a sync.Once for initialization. From there I was able to see things incrementing properly without leaking FDs:

~ daniel@p50 ✔ lsof -p $(pgrep -f node_exporter) | wc -l                                                                                                                                                                                       
305                                                                                                                                                                                                                                            
~ daniel@p50 ✔ curl -s 'http://localhost:9100/metrics?collect\[\]=perf' | grep node_perf_page_faults_total                                                                                                                                     
# HELP node_perf_page_faults_total Number of page faults                                                                                                                                                                                       
# TYPE node_perf_page_faults_total counter                                                                                                                                                                                                     
node_perf_page_faults_total{cpu="0"} 27447                                                                                                                                                                                                     
node_perf_page_faults_total{cpu="1"} 27768                                                                                                                                                                                                     
node_perf_page_faults_total{cpu="2"} 30826                                                                                                                                                                                                     
node_perf_page_faults_total{cpu="3"} 27146                                                                                                                                                                                                     
node_perf_page_faults_total{cpu="4"} 20667                                                                                                                                                                                                     
node_perf_page_faults_total{cpu="5"} 17671                                                                                                                                                                                                     
node_perf_page_faults_total{cpu="6"} 21991                                                                                                                                                                                                     
node_perf_page_faults_total{cpu="7"} 27716                                                                                                                                                                                                     
~ daniel@p50 ✔ lsof -p $(pgrep -f node_exporter) | wc -l                                                                                                                                                                                       
305                                                                                                                                                                                                                                            
~ daniel@p50 ✔ curl -s 'http://localhost:9100/metrics?collect\[\]=perf' | grep node_perf_page_faults_total                                                                                                                                     
# HELP node_perf_page_faults_total Number of page faults                                                                                                                                                                                       
# TYPE node_perf_page_faults_total counter                                                                                                                                                                                                     
node_perf_page_faults_total{cpu="0"} 27953                                                                                                                                                                                                     
node_perf_page_faults_total{cpu="1"} 28097                                                                                                                                                                                                     
node_perf_page_faults_total{cpu="2"} 31501                                                                                                                                                                                                     
node_perf_page_faults_total{cpu="3"} 28217                                                                                                                                                                                                     
node_perf_page_faults_total{cpu="4"} 21789                                                                                                                                                                                                     
node_perf_page_faults_total{cpu="5"} 18465                                                                                                                                                                                                     
node_perf_page_faults_total{cpu="6"} 22425                                                                                                                                                                                                     
node_perf_page_faults_total{cpu="7"} 28083    

[SuperQ]

A couple of documentation items:

• Please add a [FEATURE] to the CHANGELOG.md.
• We should document the correct minimum permissions needed to enable this. I'm doing some testing to see what that is.

I would add something like this to the README.md:

The perf collector may not work by default on all Linux systems due to kernel security settings. To allow access, set the following kernel sysctl.

sysctl -w kernel.perf_event_paranoid=X

See the [upstream docs](link here).

[hodgesds]

👍 I pushed up some doc changes and cleaned up the commit history.

[discordianfish]

Would be nice to refactor this a bit, maybe by using a map and loop over it like we do in similar cases in other collectors.

[hodgesds]

I made a couple of attempts at this but creating the map becomes rather difficult without using reflection because not all struct fields may be present. I can git it another attempt and it might save a little of the redundant checks, but would probably be slower. What do you think?

[hodgesds]

I think I got it working pretty well now.

[discordianfish]

@hodgesds Can you address the remaining comments?

[hodgesds]

Updated, let me know what you think. In the future I'd like to add support for kprobes but that requires a more thought into configuration if you have any ideas (does it make sense to have a config file?).

[discordianfish]

Some changes. Beside that, I still think we could do better to avoid repetition but I think it's fine for now.

[SuperQ]

Ping, please rebase this.

[hodgesds]

Added a test that will skip if perf_event_paranoid is not properly set.

[discordianfish]

Okay, looks good to me. We have worse cases of repetition and I know it's tricky.

[SuperQ]

LGTM

[agolomoodysaada]

Are these new metrics enabled by default? are there docs written somewhere for this new feature?

[hodgesds]

Are these new metrics enabled by default? are there docs written somewhere for this new feature?

See the README.