Fully xapian powered catalog search #488
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #488 +/- ##
==========================================
+ Coverage 63.18% 64.39% +1.21%
==========================================
Files 50 50
Lines 3504 3556 +52
Branches 1773 1816 +43
==========================================
+ Hits 2214 2290 +76
+ Misses 1288 1264 -24
Partials 2 2
Continue to review full report at Codecov.
|
veloman-yunkan
force-pushed
the
fully_xapian_powered_catalog_search
branch
4 times, most recently
from
9f307b3
to
d63ca4a
Compare
veloman-yunkan
changed the title
This was referenced Apr 14, 2021
veloman-yunkan
force-pushed
the
fully_xapian_powered_catalog_search
branch
from
d63ca4a
to
1d8e710
Compare
|
Incorporated @maneeshpm's fix of openzim/libzim#534 into the commits titled "Handling of non-words in publisher query" and "Catalog filtering by creator works via Xapian" (respectively commits 7bfd050 and 109e54d in this revision of the PR). |
veloman-yunkan
force-pushed
the
fully_xapian_powered_catalog_search
branch
from
1d8e710
to
459da6a
Compare
|
@mgautierfr It is important this PR is reviewed so @veloman-yunkan can move on. |
mgautierfr
reviewed
Apr 21, 2021
@mgautierfr And what about the question raised in the description of the PR? |
mgautierfr
approved these changes
Apr 27, 2021
|
@veloman-yunkan Please rebase on top of master (and merge yourself if I took too much time to come back on this PR) |
Now the LibraryTest.filterCheck unit-test validates the actual entries returned by `Library::filter` (previously only the count of the results was checked).
This diff is easier to view if whitespace change is ignored.
This should have been done back in PR #460
Moved the `filter.hasQuery()` check inside `buildXapianQuery()`. `Library::filterViaBookDB()` only cares if the query that is going to be run on the book DB would match all documents. The rest of changes related to enhancing the usage of Xapian for the catalog search will happen inside `buildXapianQuery()` and `updateBookDB()`.
This change fixes the failure of the LibraryTest.filterByPublisher unit-test broken by the previous commit. The previous approach used in `publisherQuery()` for building a phrase query enforcing the specified prefix for all terms fails if 1. the input phrase contains a non-word term that Xapian's query parser doesn't like (e.g. a standalone ampersand character, 1/2, a#1, etc); 2. the input phrase contains at least three terms that Xapian's query parser has no issue with. Using the `quest` tool (coming with xapian-tools under Ubuntu) the issue can be demonstrated as follows: ``` $ quest -o phrase -d some_xapian_db "Energy & security" Parsed Query: Query((energy@1 PHRASE 11 Zsecur@2)) Exactly 0 matches MSet: $ quest -o phrase -d some_xapian_db "Energy & security act" UnimplementedError: OP_NEAR and OP_PHRASE only currently support leaf subqueries $ quest -o phrase -d some_xapian_db 'Energy 1/2 security act' UnimplementedError: OP_NEAR and OP_PHRASE only currently support leaf subqueries $ quest -o phrase -d some_xapian_db "Energy a#1 security act" UnimplementedError: OP_NEAR and OP_PHRASE only currently support leaf subqueries ``` The problem comes from parsing the query with the default operation set to `OP_PHRASE` (exemplified by the `-o phrase` option in above invocations of `quest`). A workaround is to parse the phrase with a default operation of `OP_OR` and then combine all the terms with `OP_PHRASE`. Besides stemming should be disabled in order to target an exact phrase match (save for the non-word terms, if any, that are ignored by the query parser).
Catalog filtering by titles/description was sensitive to diacritics present in the query string. Fixed that. Also enhanced the unit test to validate the insensitivity to diacritics present in either the title/description or the query string.
Catalog filtering should now be case/diacritics insensitive for all fields. However it is not validated for language, name and category fields, and is validated for tags, creator & publisher only for text supplied in the filter (but not for values read from the book).
The library set up by LibraryTest now contains two valid books initialized via XML. Therefore XmlLibraryTest is not needed as a separate test suite.
veloman-yunkan
force-pushed
the
fully_xapian_powered_catalog_search
branch
from
845dfda
to
63e9a09
Compare
@mgautierfr I don't have enough privileges in this repository |
Wow, I will see with to change that. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #484 with the exception of the
local,remote,validandmaxSizefiltering criteria. Those can be processed via Xapian too, but is it worth the effort (though we will gain some uniformity in how a query is fulfilled - no postprocessing throughFilter::accept()will be needed)?Also
Library::filter()is greatly enhanced