Kibana replies with 400

[faust64]

As I could not find any Kibana metrics in my prometheus, I realized the exporter fails to gather its metrics:

$ kubectl logs -n logging kibana-kube-5d57f6fd4c-64rpt -c exporter
2020/11/22 17:19:34 using Kibana URL: http://localhost:5601
2020/11/22 17:19:34 Kibana username or password is not provided, assuming unauthenticated communication
2020/11/22 17:19:34 starting metrics server at :9684
2020/11/22 17:20:54 error while scraping metrics from Kibana: invalid response from Kibana status: 400 Bad Request
2020/11/22 17:21:24 error while scraping metrics from Kibana: invalid response from Kibana status: 400 Bad Request
2020/11/22 17:21:54 error while scraping metrics from Kibana: invalid response from Kibana status: 400 Bad Request

We can find those 400 in Kibana logs:

kubectl logs -n logging kibana-kube-5d57f6fd4c-64rpt -c kibana | grep ' 400 '
[...]
{"type":"response","@timestamp":"2020-11-22T17:24:54Z","tags":["api"],"pid":6,"method":"get","statusCode":400,"req":{"url":"/api/status?extended=","method":"get","headers":{"host":"localhost:5601","user-agent":"Go-http-client/1.1","accept":"application/json","accept-encoding":"gzip"},"remoteAddress":"127.0.0.1","userAgent":"Go-http-client/1.1"},"res":{"statusCode":400,"responseTime":56,"contentLength":9},"message":"GET /api/status?extended= 400 56ms - 9.0B"}
{"type":"response","@timestamp":"2020-11-22T17:25:24Z","tags":["api"],"pid":6,"method":"get","statusCode":400,"req":{"url":"/api/status?extended=","method":"get","headers":{"host":"localhost:5601","user-agent":"Go-http-client/1.1","accept":"application/json","accept-encoding":"gzip"},"remoteAddress":"127.0.0.1","userAgent":"Go-http-client/1.1"},"res":{"statusCode":400,"responseTime":99,"contentLength":9},"message":"GET /api/status?extended= 400 99ms - 9.0B"}
{"type":"response","@timestamp":"2020-11-22T17:25:54Z","tags":["api"],"pid":6,"method":"get","statusCode":400,"req":{"url":"/api/status?extended=","method":"get","headers":{"host":"localhost:5601","user-agent":"Go-http-client/1.1","accept":"application/json","accept-encoding":"gzip"},"remoteAddress":"127.0.0.1","userAgent":"Go-http-client/1.1"},"res":{"statusCode":400,"responseTime":75,"contentLength":9},"message":"GET /api/status?extended= 400 75ms - 9.0B"}

Is there something I should activate in Kibana?
Running from docker.elastic.co/kibana/kibana:7.10.0

Without the ?extended param, the /api/status route returns with:

{"name":"kibana","uuid":"4d9a609d-75aa-4f66-b13b-5a44319c2260","version":{"number":"7.10.0","build_hash":"1796b5ec8fa1e60ccea63f2e5c25ccc665b92fdc","build_number":35949,"build_snapshot":false},"status":{"overall":{"since":"2020-11-22T17:22:07.688Z","state":"green","title":"Green","nickname":"Looking good","icon":"success","uiColor":"secondary"},"statuses":[{"id":"core:[email protected]","message":"Elasticsearch is available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"core:[email protected]","message":"SavedObjects service has completed migrations and is available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"},{"id":"plugin:[email protected]","message":"All dependencies are available","since":"2020-11-22T17:22:07.688Z","state":"green","icon":"success","uiColor":"secondary"}]},"metrics":{"last_updated":"2020-11-22T17:22:03.757Z","collection_interval_in_millis":5000,"os":{"platform":"linux","platformRelease":"linux-4.19.0-12-amd64","load":{"1m":0.64501953125,"5m":0.39599609375,"15m":0.26513671875},"memory":{"total_in_bytes":12296331264,"free_in_bytes":5623910400,"used_in_bytes":6672420864},"uptime_in_millis":24802000,"distro":"CentOS","distroRelease":"CentOS-8.2.2004","cpuacct":{"control_group":"/","usage_nanos":14084976608},"cpu":{"control_group":"/","cfs_period_micros":100000,"cfs_quota_micros":130000,"stat":{"number_of_elapsed_periods":606,"number_of_times_throttled":12,"time_throttled_nanos":274125707}}},"process":{"memory":{"heap":{"total_in_bytes":161497088,"used_in_bytes":142625224,"size_limit":1526909922},"resident_set_size_in_bytes":238989312},"pid":6,"event_loop_delay":1.1287220008671284,"uptime_in_millis":152557},"response_times":{"avg_in_millis":13,"max_in_millis":13},"concurrent_connections":0,"requests":{"disconnects":0,"total":1,"statusCodes":{"200":1},"status_codes":{"200":1}}}}

[mindw]

@faust64 This is most likely caused by elastic/kibana#76054 .
The ?extended param works with 7.9.x - looks like this API (which is undocumented) is being revised. elastic/kibana#41983

Hope this helps!

[chamilad]

Thanks! Released with v7.10.x.1