Allow loading only selected projects for better compatibility in container environments

[juhoinkinen]

The memory usage of Annif can be high when Annif is used to serve many projects. For example, currently Annif itself and the YSO projects of Finto AI service take nearly 32 GB. Adding the YKL, Thema and Kauno projects (at the test instance ai.dev.finto.fi) increases memory usage to nearly 36 GB.

In container environments it is preferred to have many small containers instead of one big. Small containers help in distributing resources demand across available worker machines.

Also the Annif instance for Finto AI currently takes 3-5 mins to start as all projects are being loaded. The startup time could be reduced if it could be parallelized to different processes (of which each would need to load only some of the projects).

So, it would be valuable if instead one Annif process that loads all projects there could be many Annif processes of which each loads a subset of the projects. The API requests using a specific project (/projects/{project_id}/suggest, /projects/{project_id}/suggest-batch, /projects/{project_id}/learn) would then need to be directed to the Annif process having loaded the requested project, which could be achieved e.g. with NGINX.

For the requests to the project listing method (/projects) it is necessary for the responding Annif process to have access to the information of all projects, but not that the process has loaded all projects.

I think there could be a mechanism in Annif which would make it to load only selected projects. The selection could work via an environment variable, which would make Annif to load only those projects whose project ids match the pattern
in the env. Then in different containers one could set the env to values *-fi, *-sv and *-en to separate the YSO projects by language. Triton uses this kind of mechanism via command-line option.

(I thought there was an issue for this in Finto AI repository but did not find it.)

[juhoinkinen]

I tested this with branch issue733-allow-loading-selected-projects. For deploying one pod per projects set in OpenShift I used range loop functionality of Helm charts with keys and values. For simplicity I first grouped the projects by vocabulary (YSO, KAUNO, Thema, YKL), so four pods were created, and for each ANNIF_PROJECTS_INIT environment variable got different value (yso-*, kauno-*, thema-*, ykl-*) to load only the projects matching the pattern.

Also for adjusting the nginx configuration the Helm range loop could be used (the non-project-specific requests can be passed to any pod with round-robin distribution, and project-specific requests to the pod having the project in question loaded).

This works with surprisingly small amount of effort. (Note to self: running helm upgrade with changed nginx config does not put the new config in use; the nginx pod needs to be restarted manually...)

I used fnmatch for the matching the project ids to the environment variable. However, in real use the grouping only by vocabulary is suboptimal, because the projects of some vocabularies can consume very little memory (e.g. YKL projects take less than 500 MB) while others can consume very much more (YSO, 27 GB). I think the projects should be grouped to even out the memory consumption, while not making very many pods.

When grouping the YSO projects also with language, there are 6 pods in total (by using envs yso-*-fi, yso-*-sv, yso-*-en), which seems still reasonable.

Unfortunately fnmatch does not allow using curly braces for multiple match patterns, which would allow more freedom in grouping projects (like using {thema-*,ykl-*}). Then again, using regexps for project id matching feels too complicated.

It should be noted, that while projects can be separated to different pods, all pods must run on the same node (worker machine), because the projects data volume cannot be mounted to multiple nodes at the same time (currently, this will hopefully change in the future).

[juhoinkinen]

Actually because nginx uses regexps in location directive, it would simplify the helm-chart configuration if the env for project loading was also work as regexp (the same value could be used in both places).

In values.yaml the setting coud be as follows:

  projectGroups:
    "yso-fi": "yso-.*fi"
    "yso-sv": "yso-.*sv"
    "yso-en": "yso-.*en"
    "kauno": "kauno-.*"
    "thema": "thema-.*"
    "ykl": "ykl-.*"

[juhoinkinen]

The environment variables used by Annif could be gathered in a list somewhere in the Wiki. Now there are these envs (last two are aimed only for development):

• ANNIF_PROJECTS
• ANNIF_PROJECTS_INIT
• ANNIF_DATADIR
• ANNIF_CONFIG
• ANNIF_SETTINGS

The list could mention or link to documentation of envs of dependencies, like to envs of TensorFlow and of Hugging Face Hub.