Backend: Maui

The HTTP backend can be used to access an external automated indexing service that provides an API similar to the Annif REST API. The main use of this backend is to integrate Annif with MauiService, which is a microservice wrapper around the Maui automated indexing tool. Maui was originally created by Alyona Medelyan and described in her PhD thesis "Human-competitive automated topic indexing". We will use a forked, enhanced version of Maui and the microservice wrapper MauiService that were developed by Spatineo Inc. for the National Library of Finland.

Maui is very good at detecting topics of text based on comparing terms in a controlled vocabulary to terms that appear in the document text. However, it cannot detect more abstract topics whose labels do not appear in text. For example, a topic such as "local history" would not be suggested for a document that describes the history of a village, unless that phrase is used in the document itself. Thus Maui works best when combined with another algorithm that relies on statistical associations.

Configuration of the http backend is rather simple, but MauiService has to be set up separately.

Example configuration for Annif

[maui-en]
name=Maui English
language=en
backends=http
endpoint=http://localhost:8080/maui/jyu-eng/analyze
vocab=yso-en

Setting up MauiService

Maui is a Java application and MauiService is a servlet designed to run within a servlet container such as Apache Tomcat, so you will need to install these first. On Ubuntu 16.04 and 18.04, you can install the Java environment and Tomcat like this:

apt install tomcat8

The Maui and MauiService versions developed at the National Library of Finland also support a Voikko-based Finnish language "stemmer" (actually a lemmatizer) called FinnishStemmer, which relies on the libvoikko native library. It can be installed from a Ubuntu package:

apt install libvoikko1

Installing Maui and MauiService

The easiest way to install Maui and MauiService is to download the pre-built packages from Maven Central. You should download the newest maui JAR-with-dependencies and newest mauiservice WAR.

Preparing a vocabulary for Maui

Maui requires the vocabulary to be in a SKOS file that uses RDF/XML syntax. As an example, YSO is available in this format from Finto.fi (always the current version) or from the Annif-corpora repository (a specific frozen version).

Creating a model for Maui

Maui models are built from a small collection of indexed documents using the MauiModelBuilder class which can be executed from the command line. You also need a SKOS vocabulary (see above) and some language-specific settings (language code, stemmer and stopwords). Here is an example for building a model from Finnish language documents. Note that building models can be quite memory-intensive; here we give the Java process 4GB of memory so it won't run out.

java -Xmx4G -cp maui.jar com.entopix.maui.main.MauiModelBuilder -l ../Annif-corpora/fulltext/jyu-theses/fin-maui-train/ -m jyu-fin -v ../Annif-corpora/vocab/yso-skos.rdf -f skos -i fi -s StopwordsFinnish -t FinnishStemmer

Configuring MauiService

Usage

Load a vocabulary:

annif loadvoc maui-en /path/to/Annif-corpora/vocab/yso-en.tsv

Training the model on the Annif side is not necessary. However, a Maui model needs to be built for MauiService, see below.

Test the model with a single document:

cat document.txt | annif analyze maui-en

Evaluate a directory full of files in fulltext document corpus format:

annif eval maui-en /path/to/documents/