[hibernate-search] Introduce Hibernate Search framework and implement indexing page #6218

matthias-ronge · 2024-09-04T14:58:37Z

Issue #5760 2a) and 2b)

Follow-up pull request to #6209 (immediate diff)

The three numbers before the slash in “Indexed entries” represent the number of objects that Hibernate has already loaded from the database, the number of objects that have been prepared as indexable documents (JSONs), and finally the number of indexed documents.

Basic experience: Hibernate Search and lazy loading don't mix. It looks like we have to accept that. As a result, I have deactivated lazy loading wherever the number of members of a set is typically small (< 25). This affects most sets, e.g. projects of a template, tasks, users or properties of a template or a process, etc. If the set can typically be large (> 1000), the elements of the set are not indexed. Example: Processes of a batch. Consideration: If the number of subelements to be indexed in an object is very large, the findability of the object approaches infinity (it becomes increasingly likely that it will be found with any search query). Such indexing also makes the index enormously large. Therefore, it can be considered justifiable not to index these fields.

Kitodo/src/main/resources/log4j2.xml

henning-gerhardt · 2024-09-09T11:48:47Z

@matthias-ronge : a hopefully short general question: is it possible to use different indices with Hibernate-Search? Currently this is possible through different values with the elasticsearch.index configuration. Is this or something similar still possible? I'm asking because I'm working with different Kitodo.Production versions which has separated meta data directories on my local file system, different databases in a MariaDB database and different search prefixes in a ElasticSearch instance. This must not working in the current state of the changes nor is this a current goal but maybe something for later?

matthias-ronge · 2024-09-09T13:02:06Z

is it possible to use different indices with Hibernate-Search?

The index names for the individual objects are contained in the annotations as a string. I cannot estimate whether it is even possible to use variables here, or whether these have to be hard-coded strings at compile time; but I suspect the latter. Index access is controlled via properties such as port. You could install several index services on different ports and set the port at runtime before the program starts, or change the index data directory (as a symbolic link).

Such a feature is currently not in the scope of our development.

henning-gerhardt · 2024-09-09T13:22:28Z

Thank you @matthias-ronge for the explanation. I know and I did not expect that this usage scenario is part of the current development to use different hibernate search indices.

Edit: Maybe indexlayout-strategy-custom is a way to archive this. But this is nothing for now.

henning-gerhardt · 2024-09-13T10:40:15Z

Kitodo-DataManagement/hibernate.properties

+hibernate.search.enabled=true
+hibernate.search.backend.hosts=localhost:9200
+hibernate.search.backend.protocol=http


Question must this content not be added to the existing hibernate.cfg.xml file or did we need two configuration files?

As I understand it, this is the configuration file for the Hibernate Search framework. I would be surprised if we could mix the two configurations.

Does this mean, that everyone must run ElasticSearch / Opensearch on localhost and port 9200? If so I'm unable to do this in my development system nor on a productive environment.

henning-gerhardt · 2024-09-13T10:44:14Z

Kitodo-DataManagement/src/main/java/org/kitodo/data/database/beans/Property.java

@@ -31,9 +32,11 @@
 @Table(name = "property")


Is an @Indexed(index = "kitodo-property") annotation not missing like in the other bean files?

My understanding is that the annotation only applies to objects to be indexed. Standalone properties are not indexed as separate objects.

I understand.

henning-gerhardt · 2024-09-13T10:44:48Z

Kitodo-DataManagement/src/main/java/org/kitodo/data/database/beans/Folder.java

Is an @Indexed(index = "kitodo-folder") annotation not missing like in the other bean files?

henning-gerhardt · 2024-09-13T10:45:15Z

Kitodo-DataManagement/src/main/java/org/kitodo/data/database/beans/User.java

@@ -26,19 +26,23 @@
 import javax.persistence.OneToMany;
 import javax.persistence.Table;

+import org.hibernate.search.mapper.pojo.mapping.definition.annotation.GenericField;
 import org.kitodo.data.database.persistence.UserDAO;

 @Entity
 @Table(name = "user")


Is an @Indexed(index = "kitodo-user") annotation not missing like in the other bean files?

Users are not indexed because otherwise you cannot log in before the index is created. Then you cannot create the index because you cannot log in. This was not planned at the very beginning but is now the case.

I see. One of the chicken and egg problems at least and even an issue of privacy restrictions.

henning-gerhardt · 2024-09-13T10:46:44Z

Kitodo-DataManagement/src/test/resources/hibernate.properties

+hibernate.search.enabled=true
+hibernate.search.backend.hosts=localhost:9205
+hibernate.search.backend.protocol=http


See my comment above at the first hibernate.properties file.

henning-gerhardt · 2024-09-13T10:54:32Z

Kitodo/src/main/java/org/kitodo/production/services/index/ServerConnectionChecker.java

+ */
+class ServerConnectionChecker implements Runnable {
+    private static final Logger logger = LogManager.getLogger(ServerConnectionChecker.class);
+    private static final Pattern PATTERN_SERVER = Pattern.compile("cluster_name\\W+([^\"]*).*?number\\W+([^\"]*)",


Is this a pattern which is working with OpenSearch and ElasticSearch servers?

I don't know yet, right now I'm focused on getting the existing code to work. If not, it could be extended. I assume both are the same, but I haven't tried it yet. First, it's focused on finishing the minimal development.

Ok, fine for me and hopefully it is working without a change.

henning-gerhardt · 2024-09-13T10:56:18Z

Kitodo/src/main/resources/hibernate.cfg.xml

@@ -37,6 +37,9 @@
        <property name="hibernate.connection.verifyServerCertificate">false</property>
        <property name="hibernate.connection.useSSL">false</property>

+        <!-- Hibernate search -->


Are here the other Hibernate Search parameters are missing like used URI, port, ... which are added in the hibernate.properties file?

I can confirm that I also notice the similarity. Needs testing.

It would be nice if the Hibernate-Search properties are stored in one place / file. If this is not possible it would be bad at least for me.

henning-gerhardt · 2024-09-13T10:57:23Z

Kitodo/src/test/java/org/kitodo/production/services/command/CommandServiceTest.java

@@ -158,7 +158,7 @@ public void runNotExistingScriptAsync() throws InterruptedException {
        String commandString = scriptPath + "not_existing_script" + scriptExtension;
        CommandService service = new CommandService();
        service.runCommandAsync(commandString);
-        Thread.sleep(1000); // wait for async thread to finish;
+        Thread.sleep(2000); // wait for async thread to finish;


Why this and the lines below are changed from 1 to 2 seconds? Is this really needed?

Yes, my development machine is sometimes pretty slow and then the build aborts because of an error here. Maybe this should be handled completely differently than just waiting an arbitrary amount of time, but that would be something for a separate branch.

I understand. But this will increase the build and test time for everyone. But fine for now.

henning-gerhardt · 2024-09-13T10:57:48Z

Kitodo/src/test/resources/hibernate.properties

+hibernate.search.enabled=true
+hibernate.search.backend.hosts=localhost:9205
+hibernate.search.backend.protocol=http


See my comment on the first hibernate.properties file.

henning-gerhardt · 2024-09-13T10:58:12Z

Kitodo/src/test/resources/selenium/resources/hibernate.properties

+hibernate.search.enabled=true
+hibernate.search.backend.hosts=localhost:9205
+hibernate.search.backend.protocol=http


See my comment on the first hibernate.properties file.

matthias-ronge changed the base branch from master to hibernate-search September 4, 2024 14:59

matthias-ronge force-pushed the 5760_2a+b branch from babe208 to a2724c0 Compare September 6, 2024 08:20

henning-gerhardt reviewed Sep 6, 2024

View reviewed changes

Kitodo/src/main/resources/log4j2.xml Outdated Show resolved Hide resolved

matthias-ronge force-pushed the 5760_2a+b branch 2 times, most recently from ecbe5bb to 103a456 Compare September 6, 2024 12:33

matthias-ronge force-pushed the 5760_2a+b branch 2 times, most recently from 3d33e50 to 6f8657d Compare September 10, 2024 07:57

matthias-ronge mentioned this pull request Sep 11, 2024

[hibernate-search] Part 1: Remove custom ElasticSearch integration #6209

Merged

matthias-ronge force-pushed the 5760_2a+b branch 2 times, most recently from ec00505 to 945cc24 Compare September 12, 2024 07:29

matthias-ronge marked this pull request as ready for review September 12, 2024 07:57

solth and others added 16 commits September 13, 2024 11:37

Add HibernateSearch dependencies to DataManagement module

f94697a

Declare Hibernate Search version in root POM, and bump to 6.2.4.Final

43dbc44

Add HibernateSearch annotations to base indexed classes

10ebf2f

Add index names, add 'Indexed' annotation

adcd4a1

Add annotations for complex fields

d1a5c97

Add annotations for Docket, Filter, Ruleset and Workflow

e8c841a

Show the indexing page if the search server is available

6c28a24

Remove Create Mapping and Delete Index buttons (henceforth implied)

40e5249

Remove button to index remaining - not supported by Hibernate Search

e97eba0

Index all objects of given 'objectType' with massIndexer

26280cf

Re-implement indexing page

faeed43

Fix checkstyle

40ef313

Improve wording, add Javadoc

11bcf85

Don't show a total of 0 objects when starting indexing

caf4bae

Returns result processing to the calling class

5b2e0ef

Remove test for mapping - created transparently

d6ae013

matthias-ronge added 14 commits September 13, 2024 11:37

Set number of database objects

24a2005

Add template count

5fa7f85

Bring OpenSearch background instance for tests

732997c

Add MockDatabase index to Kitodo - DataManagement

ea5f0e3

Fix test

6c0eecc

Increase timeout (slow laptop)

5fa2d88

Fix problems

60bcacf

Fix search for ID

d97ac6a

Log all queries

9c20c89

Add missing file for tests

0c8374c

Fix tests

afd9ae8

Add Hibernate Search config file to Selenium resources

3369fa3

Remove unused imports

963e1c7

Add tasks to processes to enable sorting by sortHelperStatus

c1bbea7

matthias-ronge force-pushed the 5760_2a+b branch from 945cc24 to c1bbea7 Compare September 13, 2024 09:38

henning-gerhardt reviewed Sep 13, 2024

View reviewed changes

solth requested review from thomaslow and oliver-stoehr October 8, 2024 12:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[hibernate-search] Introduce Hibernate Search framework and implement indexing page #6218

[hibernate-search] Introduce Hibernate Search framework and implement indexing page #6218

matthias-ronge commented Sep 4, 2024 •

edited

Loading

henning-gerhardt commented Sep 9, 2024

matthias-ronge commented Sep 9, 2024

henning-gerhardt commented Sep 9, 2024 •

edited

Loading

henning-gerhardt Sep 13, 2024

matthias-ronge Oct 7, 2024

henning-gerhardt Oct 7, 2024

henning-gerhardt Sep 13, 2024

matthias-ronge Oct 7, 2024

henning-gerhardt Oct 7, 2024

henning-gerhardt Sep 13, 2024

henning-gerhardt Sep 13, 2024

matthias-ronge Oct 7, 2024

henning-gerhardt Oct 7, 2024

henning-gerhardt Sep 13, 2024

henning-gerhardt Sep 13, 2024

matthias-ronge Oct 7, 2024

henning-gerhardt Oct 7, 2024

henning-gerhardt Sep 13, 2024

matthias-ronge Oct 7, 2024

henning-gerhardt Oct 7, 2024

henning-gerhardt Sep 13, 2024

matthias-ronge Oct 7, 2024

henning-gerhardt Oct 7, 2024

henning-gerhardt Sep 13, 2024

henning-gerhardt Sep 13, 2024

[hibernate-search] Introduce Hibernate Search framework and implement indexing page #6218

Are you sure you want to change the base?

[hibernate-search] Introduce Hibernate Search framework and implement indexing page #6218

Conversation

matthias-ronge commented Sep 4, 2024 • edited Loading

henning-gerhardt commented Sep 9, 2024

matthias-ronge commented Sep 9, 2024

henning-gerhardt commented Sep 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthias-ronge commented Sep 4, 2024 •

edited

Loading

henning-gerhardt commented Sep 9, 2024 •

edited

Loading