[Metricbeat] Add 'query' metricset for prometheus module #15177

estherk0 · 2019-12-18T06:35:37Z

Motivation

There is no way to use Prometheus Query using Prometheus collect API (/metrics, /federate)
If Metricbeat can use Prometheus query API, it would be very powerful to collect only what I need.
Refer to the following documentation: https://prometheus.io/docs/prometheus/latest/querying/api/

Purpose

Put the resultant to ElasticSearch from Prometheus HTTP query API
Can request multiple queries

How to use

Configuration

metricbeat.modules:
      - module: prometheus
        metricsets: ['query']
        period: 10s
        hosts: ["prometheus-operated:9090"]
        paths:
        - name: "cpu_usage"
          path: '/api/v1/query'
          fields:
            'query': 'sum(rate(node_cpu_seconds_total[1m]))'
        - name: "labels"
          path: '/api/v1/labels'

Result

From /api/v1/query?query=sum(rate(node_cpu_seconds_total[1m]))

{
  "@timestamp": "2019-12-17T11:57:12.612Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "8.0.0"
  },
  "ecs": {
    "version": "1.2.0"
  },
  "host": {
    "name": "lma1"
  },
  "agent": {
    "id": "1b3a4b59-a0ce-43c3-ad65-a69d4110c8ca",
    "version": "8.0.0",
    "type": "metricbeat",
    "ephemeral_id": "8ad6434e-54ad-453d-b5ce-7e0dabb72f0b",
    "hostname": "lma1"
  },
  "service": {
    "address": "prometheus-operated:9090",
    "type": "prometheus"
  },
  "event": {
    "dataset": "prometheus.query",
    "module": "prometheus",
    "duration": 143009798
  },
  "metricset": {
    "name": "query",
    "period": 10000
  },
  "prometheus": {
    "query": {
      "mem_usage": {
        "status": "success",
        "data": {
          "resultType": "vector",
          "result": [
            {
              "metric": {},
              "reconciledValue": {
                "unixtimestamp": 1576583832.753000,
                "value": "2880242405376"
              }
            }
          ]
        }
      }
    }
  }
}

Please review this and give me feedback. :)

elasticmachine · 2019-12-18T06:35:40Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2019-12-18T06:35:40Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

exekias · 2019-12-18T11:05:21Z

Thank you for opening this @jabbukka!! 🎉 . Could you please explain how would you use this besides the given examples? I try to understand the general use case.

Also I wonder, would it make sense to store only the last value? How do you plan to query this data?

estherk0 · 2019-12-19T02:37:21Z

Thank you for opening this @jabbukka!! 🎉 . Could you please explain how would you use this besides the given examples? I try to understand the general use case.

@exekias Sorry for the lack of explanation. 😥
I'm trying to create a monitoring dashboard using ElasticSearch + Kibana + Canvas onKubernetes + OpenStack Cluster environment. And Prometheus collects a lot of metric data for Kubernetes and OpenStack. Therefore, I've decided to import Prometheus' metric into ElasticSearch.
(I just want to store the query result for monitoring to Elasticsearch. It doesn't need to store all raw data...)

However, it was difficult to calculate metrics with Prometheus raw data in ElasticSearch using ES Query. Because Prometheus data model are based on time series.

For examples, Histogram data model uses histogram_quantile() function to calculate quantiles.
I believe it makes much more easier if I can use PromQL to integrates Prometheus Data.

Also I wonder, would it make sense to store only the last value? How do you plan to query this data?

the last value is representative for the last query result. i. e) cpu usage rate for last 1 minute.
It's for the real-time metric data to report, not dumping the raw data.
Below is the example for configuration.

        [...]
        period: 1m
        paths:
        - name: "cpu_usage_1m"  
          fields:
             // cluster1's cpu usage for last 1 minute
            'query': 'sum(rate(node_cpu_seconds_total{cluster="cluster1",stat~="iowait"}[1m]))'
        - name: "top5_busiest_instaces"
           fields:
             // top 5 instances
             'query': 'topk(5,sum(rate(http_requests_total[1m]))by(instance))'
        [...]

I hope this made things a bit more clear.
It also related to this issue: Metricbeat and advanced Prometheus queries

exekias · 2019-12-19T11:27:57Z

metricbeat/module/prometheus/query/data.go

+	// Vector [ <unix_timestamp>, "<query_result>" ] is not acceptable for Elasticsearch.
+	// Because there are two types in one array.
+	// So change Vector to Object { unixtimestamp: "<unix_timestamp", value: "query_result" }
+	if res.Data.ResultType == "vector" {


It would be nice to come up with a good conversion for all result types. I think we can strip down much of the info we get in the response to store only timestamp + results. What do you think about doing something like this?:

For scalar:

Store timestamp under event tiemstamp (mb.Event.Timestamp) (this will be reported as @timestamp in the final event)

Store value under prometheus.query.<query_name> (ie: prometheus.query.http_request in your example)

For string:

Store timestamp under event tiemstamp (mb.Event.Timestamp)

Store value under prometheus.query.<query_name>

For vector:

Create an event for each metric

Store timestamp under event timestamp (mb.Event.Timestamp)

Store value under prometheus.query.<query_name>

Store labels under prometheus.labels

For matrix (range vector):

Create an event for each metric and (timestamp, vlue) pairs

Store timestamp under event timestamp (mb.Event.Timestamp)

Store value under prometheus.query.<query_name>

Store labels under prometheus.labels

Looks good! I'll try to apply this.

I'm applying the suggested changes on code. Please review commit #c511e95faf

metricbeat/module/prometheus/query/data.go

ChrsMark · 2020-03-17T15:59:40Z

@jabbukka apologies for having it stalled all that time. Could you rebase it on top of the latest master so as to resolve the conflicts and see what is missing? It would be nice to have it in soon.

On metricbeat/include/list_common.go you will need to import the modules like "github.com/elastic/beats/v7/metricbeat/module/.... It should be fixed if you run again make update
On metricbeat/module/prometheus/_meta/config.yml there is a new metricset that is already added.
On metricbeat/module/prometheus/fields.go it should be fixed by running again make update

Also all imports should be defined with v7 like github.com/elastic/beats/v7/libbeat/common.

ChrsMark · 2020-03-17T16:20:49Z

metricbeat/module/prometheus/query/_meta/testdata/docs.json-expected.json

@@ -0,0 +1,32 @@
+[


All files under tesdata directory are used for testing. In this a docs.plain file is needed which then will be used as a testing input. See for example https://github.com/elastic/beats/blob/a174425cb45c406a748b290dd2ef892c731cdac7/metricbeat/module/prometheus/collector/_meta/testdata/docs.plain.

ChrsMark · 2020-03-17T16:21:55Z

metricbeat/module/prometheus/query/query_test.go

+)
+
+func TestData(t *testing.T) {
+	mbtest.TestDataFiles(t, "prometheus", "query")


is this actually doing anything? As mentioned in a previous comment a docs.plain is required to be used as input in the tests.

ChrsMark · 2020-03-17T17:26:51Z

metricbeat/module/prometheus/query/data.go

+	Vectors [][]interface{}   `json:"values"`
+}
+
+func (m *MetricSet) parseResponse(body []byte, pathConfig PathConfig) ([]mb.Event, error) {


Is there any special reason for parseResponse to be a method of Metricset? If not the following should be fine:

Suggested change

func (m *MetricSet) parseResponse(body []byte, pathConfig PathConfig) ([]mb.Event, error) {

func parseResponse(body []byte, pathConfig PathConfig) ([]mb.Event, error) {

ChrsMark · 2020-03-17T17:53:05Z

metricbeat/module/prometheus/query/config.go

+type PathConfig struct {
+	Path   string        `config:"path"`
+	Fields common.MapStr `config:"fields"`
+	Name   string        `config:"name"`


Maybe it would be better to call it ~~MetricName~~ QueryName(and ~~metric_name~~ query_name) so as to be more specific.

ChrsMark · 2020-03-17T17:53:58Z

metricbeat/module/prometheus/query/config.go

+// Validate for Prometheus "query" metricset config
+func (p PathConfig) Validate() error {
+	if p.Name == "" {
+		return errors.New("`namespace` can not be empty in path configuration")


Namespace can be confusing here. We had better call it ~~metric_name~~query_name.

ChrsMark · 2020-03-17T18:15:15Z

metricbeat/module/prometheus/query/data.go

+	if err := json.Unmarshal(body, &arrayBody); err != nil {
+		return nil, "", errors.Wrap(err, "Failed to parse api response")
+	}
+


Shouldn't we check if the query was sucessful?
Sth like:

if arrayBody.Status == "error" { return nil, "", errors.Errorf("Failed to query") }

ChrsMark · 2020-03-17T18:38:19Z

metricbeat/module/prometheus/query/config.go

+
+// Config for "query" metricset
+type Config struct {
+	Paths       []PathConfig `config:"paths"`


Maybe a name like Queries would be better here. wdyt?

ChrsMark · 2020-03-17T18:38:39Z

metricbeat/module/prometheus/query/config.go

+
+// PathConfig is used to make a API request.
+type PathConfig struct {
+	Path   string        `config:"path"`


~~Maybe Query here?~~

ChrsMark · 2020-03-17T18:38:57Z

metricbeat/module/prometheus/query/config.go

+// PathConfig is used to make a API request.
+type PathConfig struct {
+	Path   string        `config:"path"`
+	Fields common.MapStr `config:"fields"`


Maybe QueryParams?

ChrsMark · 2020-03-17T18:42:16Z

metricbeat/module/prometheus/query/query.go

+
+		events, parseErr := m.parseResponse(body, pathConfig)
+		if parseErr != nil {
+			return err


Since we are in a loop I would say that we don't have to return but to log the error and continue to next query, sth like:

if parseErr != nil { m.Logger().Debug("error parsing response for ", pathConfig.Name, ": ", parseErr) reporter.Error(errors.Wrap(err, "error parsing response")) continue }

I would do the same if FetchResponse() return error too.

* Remove array from response body. Original response body has an array (Vector type). It makes difficult to query with Elasticsearch QL. New data schema: "prometheus": { "query": { "mem_usage": { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": {}, "reconciledValue": { "unixtimestamp": 1.576751116531e+09, "value": "2947656593408" } } ] } } } }

* reference url: https://prometheus.io/docs/prometheus/latest/querying/api/#expression-query-result-formats

* Prometheus API returns "string" type for query result. But actual result type is a number. * Make it easy to use ES SQL

ChrsMark · 2020-03-18T09:33:18Z

metricbeat/module/prometheus/query/config.go

+import (
+	"errors"
+
+	"github.com/elastic/beats/libbeat/common"


Suggested change

"github.com/elastic/beats/libbeat/common"

"github.com/elastic/beats/v7/libbeat/common"

ChrsMark · 2020-03-18T09:34:33Z

metricbeat/module/prometheus/query/data.go

+	"strconv"
+	"time"
+
+	"github.com/elastic/beats/libbeat/common"


Suggested change

"github.com/elastic/beats/libbeat/common"

"github.com/elastic/beats/v7/libbeat/common"

ChrsMark · 2020-03-18T09:34:51Z

metricbeat/module/prometheus/query/data.go

+	"time"
+
+	"github.com/elastic/beats/libbeat/common"
+	"github.com/elastic/beats/metricbeat/mb"


Suggested change

"github.com/elastic/beats/metricbeat/mb"

"github.com/elastic/beats/v7/metricbeat/mb"

ChrsMark · 2020-03-18T09:35:14Z

metricbeat/module/prometheus/query/query.go

+import (
+	"io/ioutil"
+
+	"github.com/elastic/beats/libbeat/common"


Suggested change

"github.com/elastic/beats/libbeat/common"

"github.com/elastic/beats/v7/libbeat/common"

ChrsMark · 2020-03-18T09:35:24Z

metricbeat/module/prometheus/query/query.go

+	"io/ioutil"
+
+	"github.com/elastic/beats/libbeat/common"
+	"github.com/elastic/beats/metricbeat/helper"


Suggested change

"github.com/elastic/beats/metricbeat/helper"

"github.com/elastic/beats/v7/metricbeat/helper"

ChrsMark · 2020-03-18T09:35:32Z

metricbeat/module/prometheus/query/query.go

+
+	"github.com/elastic/beats/libbeat/common"
+	"github.com/elastic/beats/metricbeat/helper"
+	"github.com/elastic/beats/metricbeat/mb"


Suggested change

"github.com/elastic/beats/metricbeat/mb"

"github.com/elastic/beats/v7/metricbeat/mb"

ChrsMark · 2020-03-18T09:59:50Z

metricbeat/module/prometheus/query/query.go

+
+func (m *MetricSet) getURL(path string, queryMap common.MapStr) string {
+	queryStr := mb.QueryParams(queryMap).String()
+	return "http://" + m.BaseMetricSet.Host() + path + "?" + queryStr


Suggested change

return "http://" + m.BaseMetricSet.Host() + path + "?" + queryStr

return m.http.GetURI() + path + "?" + queryStr

In order to have the URI already set properly on module initialisation you will need to initialise Metricset like:

const ( defaultScheme = "http" ) var ( hostParser = parse.URLHostParserBuilder{ DefaultScheme: defaultScheme, }.Build() ) func init() { mb.Registry.MustAddMetricSet("prometheus", "query", New, mb.WithHostParser(hostParser), ) }

ChrsMark · 2020-03-18T13:35:20Z

@jabbukka I think that the requested changes might need a decent amount of work and testing and since we are moving forward to code freeze would you mind if I do the changes-cleanup for you on a different PR while keeping your commits ( in this, in the merge commit you will be the co-author) Let me know!

estherk0 · 2020-03-19T04:42:25Z

@ChrsMark Sure! No problem. :)

ChrsMark · 2020-03-23T12:33:27Z

Hey @jabbukka ! This feature has been merged! Here is the commit on master: 7c82034.

Thank you for proposing and implementing this feature!

elasticmachine · 2020-03-23T12:34:03Z

Pinging @elastic/integrations-platforms (Team:Platforms)

estherk0 · 2020-03-24T08:15:16Z

@ChrsMark I'm so glad to contribute it! :)

estherk0 requested a review from a team as a code owner December 18, 2019 06:35

ChrsMark added Team:Integrations Label for the Integrations team [zube]: In Review review labels Dec 18, 2019

exekias reviewed Dec 19, 2019

View reviewed changes

houndci-bot reviewed Dec 21, 2019

View reviewed changes

metricbeat/module/prometheus/query/data.go Outdated Show resolved Hide resolved

houndci-bot reviewed Dec 21, 2019

View reviewed changes

metricbeat/module/prometheus/query/data.go Outdated Show resolved Hide resolved

estherk0 requested a review from exekias January 5, 2020 04:40

andresrc assigned exekias Jan 27, 2020

andresrc added [zube]: Inbox [zube]: In Review and removed [zube]: In Review [zube]: Inbox labels Jan 27, 2020

sorantis added the enhancement label Jan 27, 2020

sorantis mentioned this pull request Feb 24, 2020

[Metricbeat] OpenMetrics Module #14982

Closed

7 tasks

ChrsMark reviewed Mar 17, 2020

View reviewed changes

estherk0 added 4 commits March 18, 2020 17:13

[Metricbeat] Add 'query' metricset for prometheus module

69b12d4

[Metricbeat] Support all expression query result formats

66284f9

* reference url: https://prometheus.io/docs/prometheus/latest/querying/api/#expression-query-result-formats

[Metricbeat] convert numeric string to integer/float type

b939211

* Prometheus API returns "string" type for query result. But actual result type is a number. * Make it easy to use ES SQL

ChrsMark reviewed Mar 18, 2020

View reviewed changes

ChrsMark mentioned this pull request Mar 19, 2020

[Metricbeat] Add 'query' metricset for prometheus module #17104

Merged

5 tasks

ChrsMark closed this Mar 23, 2020

ChrsMark added the [zube]: Done label Mar 23, 2020

zube bot removed the [zube]: In Review label Mar 23, 2020

ChrsMark added [zube]: In Review Team:Platforms Label for the Integrations - Platforms team and removed review labels Mar 23, 2020

zube bot added review Team:Platforms Label for the Integrations - Platforms team and removed Team:Platforms Label for the Integrations - Platforms team [zube]: In Review review labels Mar 23, 2020

ChrsMark mentioned this pull request Mar 23, 2020

Cherry-pick #17104 to 7.x: [Metricbeat] Add 'query' metricset for prometheus module #17178

Merged

5 tasks

andresrc removed the [zube]: Done label Mar 31, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metricbeat] Add 'query' metricset for prometheus module #15177

[Metricbeat] Add 'query' metricset for prometheus module #15177

estherk0 commented Dec 18, 2019 •

edited

Loading

elasticmachine commented Dec 18, 2019

elasticmachine commented Dec 18, 2019

exekias commented Dec 18, 2019 •

edited

Loading

estherk0 commented Dec 19, 2019 •

edited

Loading

exekias Dec 19, 2019

estherk0 Dec 20, 2019

estherk0 Dec 21, 2019

ChrsMark commented Mar 17, 2020 •

edited

Loading

ChrsMark Mar 17, 2020

ChrsMark Mar 17, 2020

ChrsMark Mar 17, 2020

ChrsMark Mar 17, 2020 •

edited

Loading

ChrsMark Mar 17, 2020 •

edited

Loading

ChrsMark Mar 17, 2020

ChrsMark Mar 17, 2020

ChrsMark Mar 17, 2020 •

edited

Loading

ChrsMark Mar 17, 2020

ChrsMark Mar 17, 2020 •

edited

Loading

ChrsMark Mar 18, 2020

ChrsMark Mar 18, 2020

ChrsMark Mar 18, 2020

ChrsMark Mar 18, 2020

ChrsMark Mar 18, 2020

ChrsMark Mar 18, 2020

ChrsMark Mar 18, 2020

ChrsMark commented Mar 18, 2020

estherk0 commented Mar 19, 2020

ChrsMark commented Mar 23, 2020

elasticmachine commented Mar 23, 2020

estherk0 commented Mar 24, 2020

	func (m *MetricSet) parseResponse(body []byte, pathConfig PathConfig) ([]mb.Event, error) {
	func parseResponse(body []byte, pathConfig PathConfig) ([]mb.Event, error) {

	"github.com/elastic/beats/libbeat/common"
	"github.com/elastic/beats/v7/libbeat/common"

	"github.com/elastic/beats/metricbeat/mb"
	"github.com/elastic/beats/v7/metricbeat/mb"

	"github.com/elastic/beats/metricbeat/helper"
	"github.com/elastic/beats/v7/metricbeat/helper"

	return "http://" + m.BaseMetricSet.Host() + path + "?" + queryStr
	return m.http.GetURI() + path + "?" + queryStr

[Metricbeat] Add 'query' metricset for prometheus module #15177

[Metricbeat] Add 'query' metricset for prometheus module #15177

Conversation

estherk0 commented Dec 18, 2019 • edited Loading

Motivation

Purpose

How to use

elasticmachine commented Dec 18, 2019

elasticmachine commented Dec 18, 2019

exekias commented Dec 18, 2019 • edited Loading

estherk0 commented Dec 19, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChrsMark commented Mar 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChrsMark Mar 17, 2020 • edited Loading

Choose a reason for hiding this comment

ChrsMark Mar 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChrsMark Mar 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChrsMark Mar 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChrsMark commented Mar 18, 2020

estherk0 commented Mar 19, 2020

ChrsMark commented Mar 23, 2020

elasticmachine commented Mar 23, 2020

estherk0 commented Mar 24, 2020

estherk0 commented Dec 18, 2019 •

edited

Loading

exekias commented Dec 18, 2019 •

edited

Loading

estherk0 commented Dec 19, 2019 •

edited

Loading

ChrsMark commented Mar 17, 2020 •

edited

Loading

ChrsMark Mar 17, 2020 •

edited

Loading

ChrsMark Mar 17, 2020 •

edited

Loading

ChrsMark Mar 17, 2020 •

edited

Loading

ChrsMark Mar 17, 2020 •

edited

Loading