Fix crashing index command when targeted directory contains subject files

[juhoinkinen]

The annif index command crashes when the targeted directory contains subject files (files with either *.tsv or *.key extensions):

annif index dummy-fi tests/corpora/archaeology/fulltext/
Traceback (most recent call last):
...
  File "/home/local/jmminkin/git/Annif/annif/corpus/subject.py", line 248, in from_string
    subject_ids.add(subject_index.by_uri(uri))
AttributeError: 'NoneType' object has no attribute 'by_uri'

The bug is due to this change done in PR #663.

I made the change because it seemed unnecessary and confusing to pass the subject index and language to DocumentDirectory when it is used only for obtaining documents to give suggestions to.

Reverting the change, i.e. passing subject index and language to DocumentDirectory is the simplest way to fix the bug, and on second thought maybe it also is better conceptually: corpora should have subject index and language defined, even if they are not used.

Also renames vars to better correspond their origin/usage: keyfile(name) -> subjfile(name).

[codecov]

Codecov Report

Patch coverage: 100.00% and no project coverage change.

Comparison is base (2d1d3ce) 99.66% compared to head (b602ea2) 99.66%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #705   +/-   ##
=======================================
  Coverage   99.66%   99.66%           
=======================================
  Files          89       89           
  Lines        6288     6295    +7     
=======================================
+ Hits         6267     6274    +7     
  Misses         21       21           

Impacted Files	Coverage Δ
annif/cli.py	100.00% <100.00%> (ø)
annif/corpus/document.py	100.00% <100.00%> (ø)
tests/conftest.py	100.00% <100.00%> (ø)
tests/test_backend.py	100.00% <100.00%> (ø)
tests/test_cli.py	100.00% <100.00%> (ø)
tests/test_corpus.py	100.00% <100.00%> (ø)
tests/test_project.py	100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[osma]

This fixes the problem for sure and the variable renamings are sensible.

However, I think another approach would also be possible. DocumentDirectory already takes a require_subjects flag, and when set to False, the subjects are not required, but they are still parsed from tsv/key files and returned if such files exist. This is wasted effort in the case of the annif index operation, because the subjects aren't going to be used anyway. So another solution would be to modify DocumentDirectory so it doesn't even check for tsv/key files if require_subjects==False. Then it would also be unnecessary to pass the subject index and language to the DocumentDirectory. If the default value of these parameters were set to None, it could be constructed like this: DocumentDirectory(directory, require_subjects=False). I think that would be even more elegant than the solution in this PR. However, I'm happy to accept either solution.

[sonarcloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
18.2% Duplication

[osma]

LGTM

[san-uh]

Dear Annif colleagues, thank you for already working on this problem. We in the DNB have recently started using Annif 0.61.0 and have now noticed that we cannot currently use an "index" command with this version, so we have to switch back to 0.60.0. Just one question: Is a bugfix release 0.61.1 planned or will the solution come with 1.0? Thanks and greetings, Sandro

[juhoinkinen]

Hi @san-uh,

This bug realized only when the targeted directory contains files with either *.tsv or *.key extension (as far as I'm aware). Does index command work for you if you remove these files from the targeted directory (if there are them)?

[san-uh]

Hello @juhoinkinen,
thank you very much for the explanation. I deleted the *.tsv files that were previously in the directory and the index command now works. So with this workaround annif 0.61.0 including index is usable. Thanks for the feedback! Sandro