multilingual SubjectIndex backed by CSV file #608

osma · 2022-08-15T05:49:55Z

This PR completes (hopefully) the switch to multilingual vocabularies (#559) started in PR #600 and continued in PRs #604 and #606. It changes the SubjectIndex so that it keeps track of labels in all available languages, not just one at a time. The index is stored on disk in a CSV file which stores the labels in different columns named e.g. label_en, label_fi, label_sv etc. It replaces the current short-lived format where separate TSV files for each language were used (subjects.en.tsv, subjects.fi.tsv etc.). It turned out to be easier and cleaner to have just a single file containing labels in all languages. CSV is a good format for this as the columns can be named in a header row, so there is some flexibility in which columns are used.

It is also possible to use this CSV format to represent multilingual vocabularies that can be loaded with the annif loadvoc command.

codecov · 2022-08-15T07:05:25Z

Codecov Report

Merging #608 (d26a5db) into master (1c2e849) will increase coverage by 0.03%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #608      +/-   ##
==========================================
+ Coverage   99.55%   99.58%   +0.03%     
==========================================
  Files          86       87       +1     
  Lines        5663     5834     +171     
==========================================
+ Hits         5638     5810     +172     
+ Misses         25       24       -1

Impacted Files	Coverage Δ
tests/test_backend_http.py	`100.00% <ø> (ø)`
tests/test_eval.py	`100.00% <ø> (ø)`
annif/backend/yake.py	`98.21% <100.00%> (ø)`
annif/cli.py	`99.63% <100.00%> (+<0.01%)`	⬆️
annif/corpus/__init__.py	`100.00% <100.00%> (ø)`
annif/corpus/document.py	`100.00% <100.00%> (ø)`
annif/corpus/skos.py	`100.00% <100.00%> (ø)`
annif/corpus/subject.py	`100.00% <100.00%> (ø)`
annif/corpus/types.py	`100.00% <100.00%> (ø)`
annif/eval.py	`100.00% <100.00%> (ø)`
... and 13 more