Improve passthrough in prometheus output and input plugins

[mmolnar]

Prometheus metrics will get mangled when passed to another telegraf instance using any of the serializers due to the loss of type. This is makes the histogram and summary metrics unusable in prometheus.

This change will store prometheus help and type as metric tags in prometheus input and restore them in prometheus output.

Let me know what you think about this change. My go knowledge is mostly superficial as my main focus is system administration, I will wellcome a code review.

[danielnelson]

The preservation of TYPE support should be working already, can you open a new issue if you are still having problems in 1.7 or newer?

I would like to add support for HELP at some points, but I don't think we should pass it through as a tag, and instead it should become a new piece of metadata that can be stored with the field, similar in many ways to #3670. This issue has been a little trickier to implement than desired, as I want to find a memory efficient way to make these optional.

[mmolnar]

Serializers do not support preservation of TYPE information so this information is lost when using socket writer/socket listener. You are right that it works inside single instance of telegraf, but when using a chain of telegraf instances then the metrics are mangled.

<prometheus input> -> <telegraf> -> <socket writer> -> [loss of TYPE] -> <socket_listener> -> <telegraf> -> <prometheus output>

In this chain the mertics in prometheus output are mangled and mainly the histogram and summary metrics are unusable inside prometheus because histogram_quantile() function expects the data to be in this format:

streaming_storage_latency_seconds_bucket{le="0.05"} 56
streaming_storage_latency_seconds_bucket{le="0.1"} 83
streaming_storage_latency_seconds_bucket{le="0.5"} 143
streaming_storage_latency_seconds_bucket{le="1"} 199
streaming_storage_latency_seconds_bucket{le="5"} 215
streaming_storage_latency_seconds_bucket{le="10"} 215
streaming_storage_latency_seconds_bucket{le="+Inf"} 215
streaming_storage_latency_seconds_sum 40.25
streaming_storage_latency_seconds_count 215

Instead the metrics in prometheus output look like this:

streaming_storage_latency_seconds_0_05 56
streaming_storage_latency_seconds_0_01 83
streaming_storage_latency_seconds_0_5 143
streaming_storage_latency_seconds_1 199
streaming_storage_latency_seconds_5 215
streaming_storage_latency_seconds_10 215
streaming_storage_latency_seconds_inf 215
streaming_storage_latency_seconds_sum 40.25
streaming_storage_latency_seconds_count 215

If you plan to add TYPE and HELP metadata to the serializers then I will look forward to this feature, I just hope it will not be too long until it is implemented.

[danielnelson]

@mmolnar I opened a couple feature requests for how I think we should go about solving this. Can you look at these and let me know if you have any feedback?