Getting started with ElasticSearch and Kibana for Geonetwork Administrators

Prerequisites

• This guide assumes that you have installed and configured ElasticSearch and Kibana and can access them via a web interface. For information on how to do this, see https://geonetwork-opensource.org/manuals/3.4.x/en/maintainer-guide/statistics/index.html.
• For this guide we will assume that you are accessing them via your localhost and the default ports. In that case:
  • ElasticSearch should be accessible at http://localhost:9200
  • Kibana should be accessible at http://localhost:8080/geonetwork/dashboards/app/kibana

Terminology

The terminologies used by ElasticSearch and Kibana can be confusing if you are new to them. See https://www.elastic.co/guide/en/elasticsearch/reference/5.6/_basic_concepts.html for the full list of ElasticSearch terms, but here's a brief list of the most important concepts in ElasticSearch and Kibana from a Geonetwork perspective:

• Cluster: one or more servers or nodes that hold your ElasticSearch index. If you're just getting started with Geonetwork then your server is a cluster.
• Node: a single server within your cluster. For a basic Geonetwork installation you have a single node.
• Document: a basic piece of information that you are indexing.
• Index: a collection of documents. For geonetwork there are three indices: records,features,searchlogs
• Mapping: a definition of how you index a document, defining the type of each field, whether a field should be indexed, the format of dates and so on.
• Index Pattern: a kibana object that defines the index you wish to explore. In Geonetwork there are 5: .dashboards, indicators, records, searchlogs, features
• Visualisation: a kibana object such as a pie chart or line graph showing a specific document from a specific index-pattern. In Geonetwork, each of the graphs or charts in the Administrator Page Statistics and Status/Content Statistics and Search Statistics Tab are visualisations.
• Dashboard: a collection of visualisations. In Geonetwork there are 3: Content Statistics, Search Statistics, INSPIRE (if enabled)

Getting around ElasticSearch and Kibana

This is not an exhaustive guide to ElasticSearch and Kibana, it's an introduction to the important features as set up in Geonetwork. Do not use it as a substitute for reading the official documentation.

ElasticSearch

Each ElasticSearch index is available via your ElasticSearch URL, with the index name appended. For example http://localhost:9200/records. This returns the mappings for the records index in json form. There are various apis that you can use to explore the index. Some useful ones are:

• _cat: Find information about your catalogue. See https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cat.html
  • _cat/health: Get information about the health of the catalogue
  • _cat/indices: List the indices in the catalogue and their health
• indexname/_search: Search a given index. See https://www.elastic.co/guide/en/elasticsearch/reference/5.6/search-search.html
  • indexname/_search?q=: Query a given index
    • q=: match all documents in the index
    • q=*&size=n: match all documents and restrict to the first n results

Kibana

In Kibana you can explore the index patterns, visualisations and dashboards set up as part of your Geonetwork installation.

• Discover: Allows you to search each index-pattern, drilling down to show the fields, the distinct values, submit queries and so on.
• Visualize: Shows all the visualisations set up
• Dashboard: Shows the different dashboards. You can edit an individual dashboard to re-position or even edit different visualisations.
• Timelion: Not used in Geonetwork
• Dev Tools: A console for sending requests to the ElasticSearch endpoint and seeing the results
• Management: Splits into 3 tabs showing index patterns, saved objects and advanced settings. Of these, index-patterns is probably the most useful for the Geonetwork Admin as it shows at a glance how each index-pattern is set up.

What to do if things go wrong

If you see an error in Geonetwork's Content or Search Statistics tabs indicating that an index-pattern cannot be found, follow these steps to debug before submitting an issue in Github:

• Find the dashboard in Kibana (eg catalogue statistics) and edit it (top right)
• Edit the visualisation that isn't working
• Look at visState (at the bottom) and see the index-field-pattern that it is complaining about
• Check the appropriate ElasticSearch index (see above for how to do this) and find that field- does it exist and what type is it?
• If it does exist and is of an appropriate type, then go to Kibana's Management/index patterns tab and choose the correct index (eg searchlogs) and refresh the field list. This should bring through the additional fields
• If it doesn't exist, then edit the visState json to use the correct type and save it.
• In either case, reloading the dashboard in Geonetwork should now show all the visualisations correctly.