Server status page

[spalger]

@skearns64 made a good point in #3033:

I like the idea of having Kibana to wait indefinitely for ES, but allowing the UI to come up and report that it is unable to communicate to ES.

Right now, the Kibana UI has a hard dependency on the server being fully initialized and elasticsearch being reachable. I don't really want that to change, but I do think we can make this better.

What if we created a super simple page that just showed log messages pertaining to startup? It would show things like "migrating config from version x to y" and "waiting for elasticsearch to be ready, checking again in 5 seconds". This page could be available at localhost:5601/_startup, and when someone visits any route before startup is complete we could 302 them to /_startup?original_url=....

Once initialization completes, a final log message "startup complete" would be logged and any user that is actively viewing the startup page (who has the original_url query string param) will be redirected to the url they originally wanted.

This will require some plumbing changes in the server, but is pretty flexible.

[rashidkpc]

++

[skearns64]

+1 to this.

I wonder if we could use a similar approach to handle the case where ES goes down (e.g. full cluster restart, broken network between K4 and ES) while Kibana stays up. If this makes sense, it might warrant another story, as it requires detection of a non-responsive ES, and some way to redirect the user, but it seems very related.

[spalger]

I like that use case a lot @skearns64. Viewing this page as more of a status page than a log viewer might help us hit both of those targets at the same time. Of course, no status page is complete without a log viewer of some sort.

change the url to: localhost:9200/_status

1. server process starts status = "initializing"
   • visitors are redirected to _status?on_active={{url}}
2. initialization completes status = "active"
   • people waiting for active will now be redirected per original example
3. Kibana looses connection to elasticsearch, status = "disconnected"
   • routes are disabled again
   • visitors are redirected to _status?on_active={{url}}
4. Elasticsearch comes back, goto step 2

[ohTHATaaronbrown]

👍

[radiumx3]

Big 👍

[spalger]

After some discussion, we decided modeling it on statuspage.io is probably a decent idea:

[radiumx3]

@spalger Could you explain please ?

[spalger]

@radiumx3 we were mostly trying to nail down how the page would look, and the info it would display.

We decided that in many ways we want it to behave just like http://www.vimeostatus.com/ or https://status.github.com/ and provide information like:

• global server status 🚦
• per module status
  • elasticsearch
  • pdf-generation
  • etc.
• some simple sparklines for this server
  • memory
  • response time
  • request count
  • etc.
• recent server events/log messages
  • "startup completed"
  • "lost connection to elasticsearch
  • "worker process died"
  • "webhook received"
  • etc.
• all of the pages data should be available in json format for monitoring systems.

[radiumx3]

Oh right i get it ! Would be awesome 👍