Influxdb Exporter: Missing TAG declared in instance_keys

[albinpopote]

Describe the bug
When I declare instance keys on collector subtemplate, not all keys are set to Influxdb tags

Environment
Provide accurate information about the environment to help us reproduce the issue.

• Harvest version: harvest version 21.05.4-1 (commit 8976370) (build date 2021-08-02T11:24:55+0200) linux/amd64
• Command line arguments used: ./bin/harvest start --foreground --verbose--collectors ZapiPerf --objects SystemNode DC-01
• OS: Debian 10.9
• Install method: dpkg install from manual package compilation
• ONTAP Version: CDOT 9.5

To Reproduce
Steps to reproduce the behavior:

Declare a subtemplate of zapiperf collector (registered on defaults.yaml zapiperf collector configuration file, only this subtemplate for test) and try to add node, uuid, model and serial as influxdb tags:

name:                     Node
query:                    system:constituent
object:                   node

instance_key:             name

counters:
  - avg_processor_busy
  - cifs_ops
  - instance_name
  - node_name
  - node_uuid
  - ontap_version
  - serial_no
  - system_id                 => systemId
  - system_model              => model
  ...

export_options:
  instance_keys:
        - node
        - uuid
        - model
        - serial_no

Launch the command :

root@XXX:/opt/harvest# ./bin/harvest start --foreground --trace --collectors ZapiPerf --objects Node DC-01

Check the logs:

5:39PM TRC goharvest2/cmd/exporters/influxdb/influxdb.go:235 > DEBUG Keys: [node, uuid] Poller=DC-01 exporter=my_influxdb
5:39PM TRC goharvest2/cmd/exporters/influxdb/influxdb.go:266 > DEBUG Key: node Poller=DC-01 exporter=my_influxdb
5:39PM TRC goharvest2/cmd/exporters/influxdb/influxdb.go:266 > DEBUG Key: uuid Poller=DC-01 exporter=my_influxdb
5:39PM TRC goharvest2/cmd/exporters/influxdb/influxdb.go:318 > rendering from:
measurement=node
tag_set=[datacenter=Lab cluster=XXX node=XXX uuid=ca2c370d-8184-11eb-b61b-d039ea030550]

DEBUG tag was added to original code to show content of instance_keys:

func (e *InfluxDB) Render(data *matrix.Matrix) ([][]byte, error) {

        ...
        if x := data.GetExportOptions().GetChildS("instance_labels"); x != nil {
                labels_to_include = x.GetAllChildContentS()
        }

        e.Logger.Trace().Msgf("DEBUG Keys: [%s]", strings.Join(keys_to_include, ", "))
	...
		
	
	// tag set
        if include_all {
                for label, value := range instance.GetLabels().Map() {
                        if value != "" {
                                m.AddTag(label, value)
                        }
                }
        } else {
                for _, key := range keys_to_include {
                        e.Logger.Trace().Msgf("DEBUG Key: %s", key)
                        if value, has := instance.GetLabels().GetHas(key); has && value != "" {
                                m.AddTag(key, value)
                        }
                }
        }
	...

When I delete the "export_options" from configuration file, it seems that program switch to "include_all" mode because all labels are included in influxdb tags :

tag_set=[cluster=XXX datacenter=Lab systemId=123456789 model=AFF-A220 node=XXX name=XXX uuid=ca2c370d-8184-11eb-b61b-d039ea030550 ontap_version=NetApp\ Release\ 9.5P11:\ Tue\ Feb\ 25\ 08:58:48\ EST\ 2020 serial=123456789]

It seems that the content of instance_keys was filtering but I don't know why....

[vgratian]

Hi @albinpopote,

The problem here is the following:

• Harvest expects that object names are unique for each collector
• object is mapped to measurement in InfluxDB, which again has to be unique

Since the ZapiPerf collector already has a subtemplate for object node (system_node.yaml), it seems like that these two objects/measurements (and their tag-sets) collide in your InfluxDB.

What you can do is rename the object in your custom subtemplate and make sure it's unique. (Sometimes it's OK to have two subtemplates for the same object, as long as you make sure that the exported labels/fields are consistent).

Also be aware that the collector will rename instance_name to the value of object.

Let us know if this helps.

[cgrinds]

Improve documentation #408

[vgratian]

@albinpopote, as a side note: it's better to register your subtemplate in custom.yaml instead of default.yaml, so you don't loose changes after next update. See updated doc.