MQTT connected, not pushing data to influxdb

[darox]

I have configured the following Influxdb in telegraf.conf:

[[outputs.influxdb]]
  ## The full HTTP or UDP URL for your InfluxDB instance.
  ##
  ## Multiple urls can be specified as part of the same cluster,
  ## this means that only ONE of the urls will be written to each interval.
  # urls = ["udp://127.0.0.1:8089"] # UDP endpoint example
  urls = ["https://fqdn"] # required
  ## The target database for metrics (telegraf will create it if not exists).
  database = "simcom7000e_test" # required
  ## Name of existing retention policy to write to.  Empty string writes to
  ## the default retention policy.
  retention_policy = ""
  ## Write consistency (clusters only), can be: "any", "one", "quorum", "all"
  write_consistency = "any"

  ## Write timeout (for the InfluxDB client), formatted as a string.
  ## If not provided, will default to 5s. 0s means no timeout (not recommended).
  timeout = "5s"
  username = "user"
  password = "pwd"
  ## Set the user agent for HTTP POSTs (can be useful for log differentiation)
  user_agent = "telegraf-srv-ubn-services-1"
  ## Set UDP payload size, defaults to InfluxDB UDP Client default (512 bytes)
  # udp_payload = 512

  ## Optional SSL Config
  # ssl_ca = "/etc/telegraf/ca.pem"
  # ssl_cert = "/etc/telegraf/cert.pem"
  # ssl_key = "/etc/telegraf/key.pem"
  ## Use SSL but skip chain & host verification
  insecure_skip_verify = true
  namepass = ["*_simcom7000e_test"]
`

And the following MQTT input config in telegraf.conf

[[inputs.mqtt_consumer]]
  name_suffix = "_simcom7000e_test"
  data_format = "json"
  ## MQTT broker URLs to be used. The format should be scheme://host:port,
  ## schema can be tcp, ssl, or ws.
  servers = ["ssl://fqdn"]
  ## MQTT QoS, must be 0, 1, or 2
  qos = 0
  ## Connection timeout for initial connection in seconds
  connection_timeout = "30s"

  ## Topics to subscribe to
  topics = [
    "simcom7000e1",
  ]
  client_id = "telegraf"
  ## username and password to connect MQTT server.
  username = "user"
  password = "pwd"
  insecure_skip_verify = true

But for some reason, Telegraf is not pushing the data to Influxdb anymore. In the Telegraf logs I can see, that Telegraf connects to the MQTT broker, but no Data is sent towards Influxdb. The MQTT message looks like this:
{"temperature":25,"airpressure":1010,"humidity":77

The strange thing is, that it was working approximately one month ago. Maybe a bug that found to one of the latest telegraf updates?

[danielnelson]

Any chance the metrics are being removed by this?

namepass = ["*_simcom7000e_test"]

The way I usually test this is by enabling a file output with no filters and running telegraf while watching stdout:

[[outputs.file]]
  files = ["stdout"]

[darox]

@danielnelson Why would the namepass remove the metrics?

I confugred
[[outputs.file]] files = ["stdout", "/etc/telegraf/metric_output.out"]
in the telegraf.conf file, but nothing happens.

[darox]

I just tested without namesuffix and now it's working, so it seems to be some kind of issue with the namesuffix. Although this was working once.

[danielnelson]

Namepass would remove the metric if the name didn't match, we can use the file output to see what the actual name is. I'm not sure if Telegraf can write to the /etc/telegraf directory though so I recommend using something like /tmp/metric_output.out.

[darox]

I still have this issue, as soon as I have two different mqtt.consumer instances with different name_suffix it stops working. If I put all topics in one mqtt.consumer instance it's working fine, but I would like to have some topics to be pushed to a specific db.

[danielnelson]

That seems like a helpful observation, have you seen any of the symptoms described in #4594?

[darox]

@danielnelson I need to have a look on the logs. But In general, twoninstances of input.mqtt should be supported right?

[danielnelson]

Yes, this should be supported.

[LeeNX]

Run into a similar problem with v1.8.0, rolling back to v1.7.4 seems to resolve the my issue.

I also have two inputs.mqtt_consumer configs. The json set seems to go offline pretty quickly.

Let me know if there is anything I can either test or look into for more useful info.

[chumbert2]

Applying the following patch on the current master
https://github.com/Brightcookie/telegraf/commit/e3c4b9d6f7e9ca4af6de8cf27a6d9454080c3545
fixes the issue for us.

See #4580

[chumbert2]

Oops. I meant #4594 where the above patch is referenced.

[danielnelson]

Can you test if this change helps: #4846 (comment)

[LeeNX]

Still have the same problem with v1.8.1 on Debian, rolled back to v1.7.4 again.

I am wondering if it might be a malformed response where JSON is expected?

[danielnelson]

@LeeNX You can try one of these builds or wait for 1.8.2 later this week which will include the fix: #4846 (comment)

[LeeNX]

Thanks @darox

This currently runs my WeatherStation, so I need time to monitor and rollback, if it does not work, so I only get to testing this weekend, unless my time machine project finally starts working!

[bigdatasunil]

I am also facing same issue, telegraf unable to consumer messages from MQTT
Below is the config file i am using
[agent]
debug = true
[[inputs.mqtt_consumer]]

servers = ["tcp://localhost:1883"]
topics = [
"telegraf/#",
]
qos = 1
username = "admin"
password = "2021"
insecure_skip_verify = true
data_format = "influx"

[[outputs.influxdb]]

urls = ["http://127.0.0.1:8086"]
database = "test"

[[outputs.file]]
files = ["stdout"]

##################### OUTPUT ##################################
2020-06-24T14:03:41Z I! Starting Telegraf 1.14.4
2020-06-24T14:03:41Z I! Loaded inputs: mqtt_consumer
2020-06-24T14:03:41Z I! Loaded aggregators:
2020-06-24T14:03:41Z I! Loaded processors:
2020-06-24T14:03:41Z I! Loaded outputs: influxdb file
2020-06-24T14:03:41Z I! Tags enabled: host=ubuntu
2020-06-24T14:03:41Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"ubuntu", Flush Interval:10s
2020-06-24T14:03:41Z D! [agent] Initializing plugins
2020-06-24T14:03:41Z D! [agent] Connecting outputs
2020-06-24T14:03:41Z D! [agent] Attempting connection to [outputs.influxdb]
2020-06-24T14:03:41Z D! [agent] Successfully connected to outputs.influxdb
2020-06-24T14:03:41Z D! [agent] Attempting connection to [outputs.file]
2020-06-24T14:03:41Z D! [agent] Successfully connected to outputs.file
2020-06-24T14:03:41Z D! [agent] Starting service inputs
2020-06-24T14:03:41Z I! [inputs.mqtt_consumer] Connected [tcp://localhost:1883]
2020-06-24T14:03:41Z D! [inputs.mqtt_consumer] Session found [tcp://localhost:1883]
2020-06-24T14:04:00Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2020-06-24T14:04:00Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2020-06-24T14:04:10Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2020-06-24T14:04:10Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2020-06-24T14:04:20Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics
2020-06-24T14:04:20Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2020-06-24T14:04:30Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
2020-06-24T14:04:30Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics

Could anyone please suggest me what is the issue here?