The process of typing into fields is too slow in a large database. Caused by sorting columns in main table.

[ThiloteE]

JabRef 5.7--2022-06-12--e3b2ab2
Linux 5.17.13-300.fc36.x86_64 amd64
Java 18.0.1
JavaFX unknown

I am quite happy with the performance, however,

• when typing into fields in the entry editor, the process is often far too slow in actually accepting the input and showing it in the field.

Originally posted by @ilippert in #5071 (comment)

[ThiloteE]

meta-issue: #8906

[ilippert]

JabRef 5.8--2022-08-16--6a71395
Linux 5.18.17-200.fc36.x86_64 amd64
Java 18.0.2
JavaFX 18.0.2+2

If I may, I would suggest this is a very serious bug - as it is likely to prevent users from actually utilising JabRef to make extensive notes on their literature, which surely is a key reason to use JabRef (imho).

[ilippert]

JabRef 5.8--2022-08-29--eff7e65
Linux 5.19.4-200.fc36.x86_64 amd64
Java 18.0.2.1
JavaFX 18.0.2+2

Work around for me: type in notes on literature in an external pure text editor, then copy & paste in JabRef. Works very well.

To me, this reveals: the problem is not the amount of text but the typing.

[Siedlerchr]

@ilippert Do you have autocomplete enabled or some save actions configured or maybe update of timestamp (modification date)?
For example to prevent syncing every keystroke with a shared database, JabRef has a kind of event filter which only triggers after a certain amount of characters is changed not sure if this is intefering here

[ilippert]

@ilippert Do you have autocomplete enabled

no.

or some save actions configured or maybe update of timestamp (modification date)?

yes, I have update timestamp modificationdate activated.

For example to prevent syncing every keystroke with a shared database, JabRef has a kind of event filter which only triggers after a certain amount of characters is changed not sure if this is intefering here

Indeed, when I deactivate modificationdate then typing is swift.

[ilippert]

JabRef 5.8--2022-09-04--f240f6d
Linux 5.19.6-200.fc36.x86_64 amd64
Java 18.0.2.1
JavaFX 18.0.2+2

However, in this version, when both autocompletion and the creationdate & modificationdate are DISABLED, typing is slow.

[Siedlerchr]

Is Autosaving enabled? I wonder if the backup manager produces these issues. But it might be also some weird java thing.

What linux os are you using again? Arch?

[ilippert]

Is Autosaving enabled? I wonder if the backup manager produces these issues. But it might be also some weird java thing.

autosaving is disabled

What linux os are you using again? Arch?

Fedora 36

[ilippert]

Meanwhile on
JabRef 5.8--2022-12-08--8ebdb52
Linux 6.0.11-300.fc37.x86_64 amd64
Java 18.0.2.1
JavaFX 19+11
and Fedora 37.

Autosaving is still disabled, as is timestamp on modification. Timestamp on creationdate is active however. The problem still appears.

seemingly, if both timestamp options are disabled, the problem does not appear.

[m-mauersberger]

Maybe this issue has something to do with large databases?

On a new database with new entries, I can type seemingly properly. But there are serious delays in a new entry of a database with >8000 entries. In a similar way as @ilippert, I am creating new entries in a new local database and copy&paste it to the large database.

[ilippert]

Now I am on
JabRef 5.10--2023-01-12--5d24847
Linux 6.0.18-300.fc37.x86_64 amd64
Java 19.0.1
JavaFX 19+11
And I have disabled Autosaving and both types of timestamp updates. But the problem is still there.

[ytzemih]

I can confirm this issue (i.e., very slow typing) whenever I'm using search for filtering the database (i.e. having some text in the search field) and edit some entries in the search result.

My solution:

1. Search
2. Select entry to be edited
3. Move cursor into the entry editor
4. Clear search field
5. Make my edits

Perhaps, programmatically, there is some way to reduce the "update dependency" by relaxing the "search update trigger" of the search field and the "update notification publisher" in the entry editor. (Please, excuse my uninformed jargon, I'm not too familiar with JR's GUI framework.

On a side note, it occurs to me that the "update notification" in the entry editor would be good to implement some delay,
because I think it is not necessary to trigger "search" and "preview" processor after each key stroke.

[Siedlerchr]

JabRef has some delay as it waits for major changes after typing: CoarseChangeFilter is responsible for this

[ytzemih]

@Siedlerchr Hmm, sounds like that filter could solve the issue. Or, are you saying this is already implemented?

In my JR installation it appears as if each key stroke triggers a new search as well as a new rendering of the preview pane. So, there is quite a bit to deal with after each stroke.

[Siedlerchr]

For searching, we only have a small delay and there is no filter, search is instant, so whenever the content of the search field changed the search is restarted

jabref/src/main/java/org/jabref/gui/search/GlobalSearchBar.java

Lines 202 to 209 in e7f55d9

	Timer searchTask = FxTimer.create(java.time.Duration.ofMillis(SEARCH_DELAY), this::performSearch);
	BindingsHelper.bindBidirectional(
	stateManager.activeSearchQueryProperty(),
	searchField.textProperty(),
	searchTerm -> {
	// Async update
	searchTask.restart();
	},

[ytzemih]

I see, ok. While this behaviour after changing content in the search bar itself is totally reasonable, typing in a field of the entry editor (e.g., changing the content of the comment field in the entry editor) should perhaps not lead to instant searches or preview renderings after each key stroke. Hope that makes sense.

[ilippert]

JabRef 5.10--2023-07-04--ffedea2
Linux 6.4.8-200.fc38.x86_64 amd64
Java 21-internal
JavaFX 20+19

no surprise, issue persists on different hardware configuration.

[ThiloteE]

Today I tried to reproduce with my 100 000 entries database and I can confirm. It is slightly slow, but not all too slow. (0.1-1 second laggs. Whatever you type will show up, albeit slightly late). I assume, because my CPU was not always completely maxed out, it is still bearable. I can confirm that the issue is caused by high CPU. RAM stays constant. Tests were done with Windows Task Manager and changing the CPU window to "logical processors". While typing, 6 of my cores (6 core cpu, 12 threads) show a significant spike:

For developers, this seems worth fixing.
For users, a workaround for now could be to upgrade the CPU to something similar or preferably better than AMD Ryzen 5 5600. See https://www.cpubenchmark.net/CPU_mega_page.html for comparisons.

[ThiloteE]

JabRef 100.0.0
Linux 5.4.0-159-generic amd64
Java 20.0.1
JavaFX 20+19

Surprisingly, with my old laptop, which features a very weak Intel Pentium N3540, the process of typing into fields feels very similar to what it feels like on my windows machine with the way more powerful Ryzen 5 5600.

This makes me believe, this is not as hardware dependent as I initially thought and better hardware may only marginally contribute to fixing this problem.

[ThiloteE]

In the newest development version from today (2023-09-06), it takes a few minutes, but then a typing into fields is lightning fast on my ryzen 5600 and with the 100 000 entries library. Something must have changed.

[ThiloteE]

Never mind! It is still slow, but I found some conditions. Here is how you can reproduce it:

1. Sort by citationkey from low to highest

   FAST: 0 --> 100 000

2. Sort by citationkey from low to high with first character priority

   Slow: 0 --> 1 --> 10 --> 100 --> 1000 --> ... --> 10009 --> 1001 --> 10010 --> ...

3. Sorty by citationkey from high to low

   Slowest: 100 000 --> 0

here is the library file:
000_hugelibraryfile100000entries.zip

[ThiloteE]

Another way how to reproduce:

1. Open two libraries. One of them has to be huge. The other can be small.
2. Sort the huge library from high --> low.
3. Switch between libraries.

Also, the time it takes to search something feels similar to the time it takes to switch libraries or to type into fields.

[ThiloteE]

Once you sort from low --> high, search stays slow, but switching between libraries and typing into fields becomes fast.

[teertinker]

Hello, I just want to report, that I am bothered by the same problem. Typing in the editor is often so slow, that it hurts. Sometimes, the cursor addionally jumps to the first position. than I have to start all over again.

For some years I kept using JabRef 4.x because of these performance issues. At some time I switched to the most current Version of JabRef because there are so many great features. I am not sure yet, wether this was a clever move. I have an ThinkPad I7 with 8GB RAM (Win10). So performance must not be a problem.

Today I accidentally found out that clearing the search field leads to a much faster process. Before that, I tried different modifications of the settings, just to get Jabref to work properly (disable: fulltextsearch, disable autocompletion, close groups -> all of these nice options, I really would like to keep).

This bug report gives some further hints (creationdate && sorting order). While I may try them for a better experience, it is definitely not a solution to create a complex set of settings just for the sake, to be able to enter characters into a textfield.

[koppor]

Today I accidentally found out that clearing the search field leads to a much faster process.

Thank you for letting us know.

The hope is then #8963 could fix something.

However, I don't really think so, because the whole performance thing is about JavaFX and we do data processing in-memory without optimizations.

[gianlucabaldassarre]

JabRef 5.11--2023-10-21--affb6ac
Windows 11 10.0 amd64
Java 21.0.1
JavaFX 20.0.2+3

I had the same issue with my JabRef terribly slowing down, and sometimes crushing, when typing in the fields.
I have a large repository containing almost 10.000 references.
The following suggestion from ytzemih, reported above ( #8977 (comment) )
solves the problem in my case (thanks ytzemih!):

1. Search adding text in the search field
2. Select entry to be edited
3. Move cursor into the entry editor
4. Clear search field
5. Make my edits

So it seems the search field is the cause of the problem (at least in my case).

I have to say that in the past I did not understand the problem for years (for some time I used the an old version of JabRef to avoid the problem). So I was very glad to discover the trick above few days ago after years of suffering! Probably I did not understand that the problem was the search field because I did not expect such a fundamental feature could cause issues.
Also in my case, I thought multiple times to pass to another App as this problem was so severe that it prevented my work (now I can say luckily I didn't as I am a big fan of JabRef! : )

I do not know if this might help. Wouldn't it be possible to add a button aside the search field so that the search is done only when pressing it, or when pressing , rather than having the continuous search? (...or any another solution, of course).

[teertinker]

I have no idea about the backend of the search engine. But wouldn't it be possible, that the search routine is only called, when there is any cursor activity in the search field?
I have three different system setups with JabRef and the problem occurs in each of them. I suppose the bug affects multiple users

[ytzemih]

Thanks, @gianlucabaldassarre for confirming the issue, which btw. is still present in a three week old snapshot.

@teertinker, as suggested in #8977 (comment), my impression really is that there is a connection between the fields in the entry editor and the search field/trigger. And, indeed, that behaviour is quite disruptive with larger databases, from a user point of view. But, certainly, it seems it is far less trivial to fix programmatically. Moreover, I suppose that a large fraction of JR users have comparatively small databases where the problem is less apparent...

In fact, I can confirm that with one database with around 200ish entries, it barely shows on my machine.

[m-mauersberger]

I am facing the same issue described in the previous comments: slow editing when search is "enabled". Thanks for the hints in
#8977 (comment) ! But the problem is handled far better in v5.11 compared to previous versions.

There were some efforts about only syncing changes in field contents that were greater than 1 (i.e. copying something or deleting characters after selecting them, see #6663). Maybe a solution could look like that?

Alternatively, a search button - that is also triggered by return key for an efficient workflow - would be a good option in my opinion as well.

[ytzemih]

Perhaps, it's too far fetched, but what about triggering "search" only when leaving the bibentry editor, e.g., by clicking on another entry in the list or on a group in the group pane? Basically "on change of focus".

Also, when editing an entry, only undesired things can happen anyway, such as the entry is taken from the list, and other entries are not even touched when not loaded in the entry editor. So, the only relevant change that can happen during editing an entry is that it's taken off the list and that is quite irrelevant, I'd guess.

[Siedlerchr]

We are currently discussing this as well and will probably remove the EntryChangedListener and see if this works better. This would prevent re-triggering the search every time. The only little drawback I see, changed entry (if it no longer matches the search, e.g. author changed from a to b) would still be shown until the search is manualy redone.

[k3KAW8Pnf7mkmdSMPHz27]

We are looking at removing the update to the search results on editing changes, and see if that resolves the performance problem.

Basically "on change of focus".

That is probably the best solution 😁
But it doesn't seem easy to do right now. If removing changes on edits resolves the performance issue, a new issue should probably be opened for that.

[ytzemih]

Thanks, @Siedlerchr and @k3KAW8Pnf7mkmdSMPHz27 , well, sounds like there is reasonable solution with some small, perhaps negligible, drawback.

[teertinker]

Dear JabRef developers. Thank you so much for being so responsive. There aren't many software projects, that take hick'ups so serious and are so fast in improving the situation.

[ytzemih]

I'd like to provide another observation that suggests this problem not to be solved entirely: So, while reducing search triggers indeed improved editing in large databases for me, interestingly, editing entries in quite small databases can still be quite slow because of the entry rendering (the pane to the right of the entry form) being triggered at each key stroke.

My uninformed guess: It seems as if there are too many things (search, entry rendering, group admin, etc.) listening too often to too many events going on on the event bus.

I'm currently using:

JabRef 5.12--2023-11-07--2afd1f6
Linux 6.2.0-36-generic amd64 
Java 21.0.1 
JavaFX 21.0.1+6

[Siedlerchr]

We merged the fix regarding the search. Thanks again all for testing and reporting! This is very important for us.
@ytzemih I have moved your hints regarding the entry preview to a new issue #10632

[ytzemih]

@Siedlerchr, my pleasure. Thanks to you all for taking care of this issue and also for opening new one based on my comment.

[ThiloteE]

I am glad one performance bottleneck seemingly has been eradicated, but the issue I described in #8977 (comment) is still there. Typing into fields is still slow when a column in the main table is sorted in reverse, hence I am reopening.

[ryan-carpenter]

The only little drawback I see, changed entry (if it no longer matches the search, e.g. author changed from a to b) would still be shown until the search is manualy redone.

True, but this can also be beneficial. For example, I add personal keywords to the groups field to document screening decisions and then use searches to locate the entries not yet ruled out. I have had some difficulty with searches being so eager that the entry I am editing disappears before I finish. Sometimes I type in another field first and then paste into groups to prevent the premature disappearance. Basically, there is a use case for keeping the search results on a temporary workbench. Having said that, I should add that I really love having a 'live' search that follows when I change from one view to another (by clicking on different groups)

[ilippert]

JabRef 5.16--2024-07-24--d8ebffc
Linux 6.9.10-200.fc40.x86_64 amd64
Java 21.0.2
JavaFX 22.0.2+4

still facing this issue

[LoayGhreeb]

JabRef 5.16--2024-07-24--d8ebffc Linux 6.9.10-200.fc40.x86_64 amd64 Java 21.0.2 JavaFX 22.0.2+4

still facing this issue

@ilippert How can I reproduce the issue?
I've enabled auto-completion and timestamp, with a library of over 6,000 entries. Typing in the entry editor is smooth.

[LoayGhreeb]

JabRef 5.16--2024-07-24--d8ebffc Linux 6.9.10-200.fc40.x86_64 amd64 Java 21.0.2 JavaFX 22.0.2+4

still facing this issue

@ilippert, could you test the build from PR #11510 and see if there is any difference?