Use the simple dictionary in fts for the user directory

[babolivier]

Some user directory searches would fail because PostgreSQL would ignore common English words. For example, when looking up a user with the display name "Bela":

$ curl -H "Authorization: Bearer [...]" https://trelawney.brendanabolivier.com/_matrix/client/r0/user_directory/search -d '{"search_term": "b"}'
{"limited":false,"results":[{"user_id":"@sometestuser:labs.abolivier.bzh","display_name":"Bela","avatar_url":null}]}
$ curl -H "Authorization: Bearer [...]" https://trelawney.brendanabolivier.com/_matrix/client/r0/user_directory/search -d '{"search_term": "be"}' 
{"limited":false,"results":[]}

In this example, it identifies "be" as a common English word and don't consider it for a full text search.

The fix to this is to use the "simple" dictionary which doesn't have any stop word, i.e. any common word that needs to be ignored when running a full text search. The result of that change in the example above is:

$ curl -H "Authorization: Bearer [...]" https://trelawney.brendanabolivier.com/_matrix/client/r0/user_directory/search -d '{"search_term": "b"}'
{"limited":false,"results":[{"user_id":"@sometestuser:labs.abolivier.bzh","display_name":"Bela","avatar_url":null}]}
$ curl -H "Authorization: Bearer [...]" https://trelawney.brendanabolivier.com/_matrix/client/r0/user_directory/search -d '{"search_term": "be"}'
{"limited":false,"results":[{"user_id":"@sometestuser:labs.abolivier.bzh","display_name":"Bela","avatar_url":null}]}

Looking up "be" now correctly yields our "Bela" user in the search results.

fwiw my initial intention is mostly to fix this for search_user_dir but I've also changed _update_profile_in_user_dir_txn that way for consistency.

[anoadragon453]

Probably a less common usecase I'll admit, but would this hamper results for displaynames that actually did have "be" as a separate word in them? Say "Be Awesome" or something.

[babolivier]

@anoadragon453 No, it would still work - this PR just makes it so "be" isn't considered a word to ignore for fts.

[clokep]

Based on the conversation on matrix this seems OK!

I think this fixes #2931, by the way.

[callahad]

Probably a less common usecase I'll admit, but would this hamper results for displaynames that actually did have "be" as a separate word in them? Say "Be Awesome" or something.

For display names with stop words ("Be Awesome"), do existing databases have bad data? Is there anything we can do to remediate that?

If I'm reading this right, update_profile_in_user_dir() was previously inserting into the user_directory_search table using an English tsvector. Would that mean stopwords like "be" get omitted, and would this be missing in the tsvector for entries populated prior to this patch?

[callahad]

(cc @babolivier for the above)

[callahad]

For example:

postgres=# SELECT to_tsvector('english', 'be awesome');
 to_tsvector 
-------------
 'awesom':2
(1 row)

postgres=# SELECT to_tsvector('simple', 'be awesome');
    to_tsvector     
--------------------
 'awesome':2 'be':1
(1 row)

And then considering this patch changes stuff like this:

diff --git a/synapse/storage/databases/main/user_directory.py b/synapse/storage/databases/main/user_directory.py
index d87ceec6da8..b4fa8a7b613 100644
--- a/synapse/storage/databases/main/user_directory.py
+++ b/synapse/storage/databases/main/user_directory.py
@@ -393,9 +393,9 @@ def _update_profile_in_user_dir_txn(txn):
                     sql = """
                         INSERT INTO user_directory_search(user_id, vector)
                         VALUES (?,
-                            setweight(to_tsvector('english', ?), 'A')
-                            || setweight(to_tsvector('english', ?), 'D')
-                            || setweight(to_tsvector('english', COALESCE(?, '')), 'B')
+                            setweight(to_tsvector('simple', ?), 'A')
+                            || setweight(to_tsvector('simple', ?), 'D')
+                            || setweight(to_tsvector('simple', COALESCE(?, '')), 'B')
                         ) ON CONFLICT (user_id) DO UPDATE SET vector=EXCLUDED.vector
                     """
                     txn.execute(

...so what about older contents of user_directory_search?