JabRef 5 on Linux x86_64 Extreme Memory Pressure, System Hangs, Unusable #5919
Comments
|
JabRef 5.0-beta.403--2020-02-03--0d42a87 Cannot confirm for current dev version of JabRef on Windows. So this problem might be Linux-specific (?). |
It is very likely Linux specific, but can you tell how big the VM is on Windows? That's my first question about that comparison. In any case, you are probably using Oracle Java, and that is unlikely on Linux desktop systems. So, there are a lot of differences that need to be disentangled before any comparison is interpretable. |
|
I guess the difference between this bug and the other two, is just that I don't notice the problem right away. I don't notice much delay until the memory issue starts pushing huge numbers of pages out of main memory. And, there is no way to guarantee this will not happen unless the binary is altered so that less that 100 GB of virtual memory is needed. I have actually built java monoliths before, but I still don't understand all of it. |
|
Hmm, does that mean, that you do not always encounter this bug? I can test this also with a Linux installation, but if it is not easy to reproduce it might get difficult. Is triggering the issue dependant on your database/database size? |
|
I updated some Java dependencies, could you please check the latest version from today? I'm just curious if it is better ( |
|
JabRef 5.0-beta.406--2020-02-05--6fda10a Still a problem. I don't understand why the "version" above says Java 13.0.2. Is that jre packaged in the deb file? My system has OpenJDK 8 and 11: Update: no, slowness of keystrokes is not such a problem as formerly. It's just when it starts up, I think. The CPU usage on four CPUs goes up briefly to 80% or 100% on startup:
then after a minute or two, things calm down a little and keystrokes are normal speed. It's a clear improvement. |
|
Hmm, I appreciate that CPU contention could cause slow downs, but that is not what I'm reporting here. At least I don't think it is, because I don't notice any CPU pressure. I'll try to get a graph, but for now, let's assume that is not this bug. My slowness is caused by swapping and memory exhaustion, but I have more than 2GB free when start jabref. |
|
I'm also seeing this: But, that is probably a different issue. |
|
Before I start Jabref 5, I have about 3GB used out of 7GB; after starting
Jabref it rises to about 4GB.
|
|
But, I'm working more carefully, now, and I need to amend some statements.
I wonder if it's just too big. it uses about 3 times as much memory as my firefox, just when I start, now it's 8 times. All I did was create one new entry. |
|
Okay, more details. I tried to finish entering the rest of that one entry and JabRef ate all of my remaining swap before I could finish. I had to quit it, but it's hung and has a bunch of memory locked. Whatever this is, it's a memory issue. I don't know why JabRef is using so much memory, but it is way to much. It's 1.3 GB in of RAM used right now and it supposed to be quitting. I don't know how to help troubleshoot this further, but I'm happy to help. Just let me know what I can do. |
|
JabRef ships with a packaged version of the JDK 13, a custom runtime image, which contains all necesary modules. If you want to run from code, you need to install a jdk 13. |
Yes, I will check ASAP. |
Ah, yes, right, that makes sense. I had trouble with the code for JabRef 5, so I was glad to find the rpm. Maybe I'll try the source if we cannot figure it out
Cool. Can I use the profiler on the packaged version or is that why you mentioned the code? |
|
OMG, I tried the latest nightly build on Sunday and it almost crashed my computer. I have 16 GB of RAM and I don't know how much of that was free, but I had over 4 GB of free swap space and after less than 5 minutes running, I was able to run three passes of the Integrity Check tool on my .bib file and then my hole computer locked up swapping. The storage light was on solid for about 60 seconds and my mouse was completely stuck. It used up nearly all of the 4 GB free swap when it finally quit swapping. 4 GB! Why is JabRef consuming huge amounts of memory? 4GB is 10 times the resident size of Firefox. Something is seriously wrong. I'd love to help, but I don't know what I can do; I really cannot afford to be crashing my system every time I test. But, let me know what I can do and I'll try. I think I will install Jab 5 on a VM, that way I can contain the damage. |
|
JabRef 5.0-beta.432--2020-02-19--c768697 Cannot confirm with the current JabRef master on Ubuntu 18.04. I am using a database which contains nearly 20,000 entries. While I do experience performance problems (see here: #5071 and here: #5735), they are quite different from what you report. My database uses 3.2 GB of RAM on a virtual machine with just 4 GB of RAM - Swap remains untouched (just 268 KB used). I do remember, however, that sometimes in the past, the same database would use up to 8 GB of RAM. It could be that this was a bug that has been fixed in the meantime, but my impression was that it depended on how you opened and interacted with the database. If you opened the database and immediately started clicking/interacting with it - without waiting for the entry preview to be fully loaded (#5735), then the RAM consumption would shoot up. Unfortunately, I was never able to replicate this behaviour in a consistent manner, so I never reported it. Similarly, if I used a database which had been generated in an older version of JabRef 3.8.2 and not yet converted/saved with the new version 5.0, performance issues would also arise (see #5735 (comment)). If none of these descriptions meet your problem, then maybe there is something going on with your database and/or Linux installation (?). If it is the latter, JabRef on a fresh VM installation might, indeed, shed light on this problem. |
|
JabRef 5.0-beta.424--2020-02-18--19bfe3e
Linux 5.3.0-40-generic amd64
Java 13.0.2
If there is a possibility that there's a connection between slowness and
previewing, then it bears noting that the preview function in 5beta is
still broken. Options/Entry Preview still shows no entries at all. #5611
[image: image.png]
|
|
@AEgit thanks for trying to test; I'm thankful for the help. However, I think you actually have confirmed this issue. What I'm seeing is lots of memory pressure, and you just confirmed that is what you saw. 3.2 GB is huge for resident memory! Remember FF is only 0.5 GB. Why would JabRef need 8 times that? My system is swapping because I'm actually using the rest of my memory for other programs. If you don't have other memory pressure, you will not swap and will not experience a slow down. But what I'm reporting is the memory use, the slowdown is just a side effect. (I didn't realize this when I opened the bug, but now I have more information.) If JabRef uses 4 GB of RAM, then that is not usable and I think it is a bug. Also, I use an RPM distribution, fedora, so Ubuntu might not be a fair test. I don't know the build system, but I downloaded a completely different file for JabRef 5 than anyone who has Ubuntu. I realize that I didn't mention this, before, so I thought I should. I'm pretty sure this is not the problem, though. Also, I have several "databases" open all the time, but the largest one is only 340 entries. So, I'm sure this has nothing to do with database size issue (#5735). Possibly multiple databases could be part of the issue, though, so let me know if you think that is significant. I still plan to VM test this, though, so I'm not backing off that promise. My VMs do not have 4 GB, so I will likely experience host slowdowns even with JabRef in the VM, if this extreme memory size is indeed standard behavior. If it is something weird about my host system, then JabRef will not use 4 GB of memory in the VM. Perhaps I can compare v4 to v5 in the VM, while I'm at it. I used 4 quite a bit before I upgraded to 5 pretty recently, and I don't remember these problems with it. So, IIRC, this is not just an unreasonable amount of memory being used by v5, but a huge increase relative to v4, which is more evidence for a bug. |
|
A little more information. It's hard to compare apples to apples, but I checked Zotero w/ 500 database entries (very few have full text) and, on Linux, I see 320 MB resident / 2.2 GB virtual size, no swap used. On Windows, it reported less than 200 MB used for the same database. Not sure why windows seems to be more memory efficient, but whatever. The points stands: the memory JabRef is using is 10x bigger than what I would expect, all other things being equal. I still haven't isolated this in my VM, but it will try it when I get a chance! |
|
I have noted some issues about releasing unused memory. I have to two databases which when loaded occupy 1.2 Gbyte memory which with 16 Gb i can afford. However, after having loaded the second, 16000 entry library and selected all entries there were 3.2 Gb used. Closing the library did not free the memory. This is wasting memory and should be most probably addressed. |
|
In case this is of interest: i have 10600 entries - without doing anything with it JabRef 5.0--2020-03-08--93de138 I have memory use of 1,5GB, VirtualMemory 107,2GB, ResidentMemory 1,6GB, Shared Memory 200MB. |
|
I still haven't tied it isolated, but that is seem less relevant as I learn more. I accidentally clicked the JabFox button, today, and JabRef started, again hanging my whole system for about a minute until 1.2 GB of memory was swapped out of RAM. I looked at other processes running and found that JabRef is much larger than most, as I previously reported, but not the largest. Virtual size was 97 GB, but that was within 2 % of the next largest. Resident Mem was 1.2 GB, which matches the swapped out memory, but there was another process that was twice that. So, I guess it is not really outside the realm of possibility. But, why? These other monster programs make some sense, and grow over time. When closed and reopened, they don't start up with this kind of pressure, I have to use them and they cache stuff. The only process with larger RES mem dropped to 255 MB after restart. That is why JabRef crushes my system, because it uses 1.2 GB on start. That part is still highly unusual. There are basically only two other "big" programs to which I can compare JabRef. One is webkit2gtk, which is related to the browser, so I kind of get it, but, even after weeks of running the same instance, the RES size is <300 kB! So, it doesn't generate nearly the pressure that JabRef does, even though it's total size is the same. And the other really big VM process also has a tiny RES size, <90 kB, after weeks of running. So, even compared to it's peers, JabRef is an outlier. I'm asking, can this be reduced, somehow? Why does JabRef load so much into memory at the beginning? |
|
JabRef 5.1--2020-03-28--64e35c1 Thinkpad T580, Core i5 8th gen, Linux Mint Cinnamon. Here's what happens to my eight CPUs when I start JabRef:
|
|
This bug may be fixed in the latest development version. Could you please check the build from http://builds.jabref.org/master/. Thanks! Please remember to make a backup of your library before trying-out this version. |
|
Okay, cool. Thanks @tobiasdiez ! I'll try it soon. |
|
JabRef 5.1--2020-04-21--dc1e339 It is awesome - I was able to run various operations at high speed (1-5 sec), that in the last test versions always took 20-40 seconds. Memory 1,4gb |
|
I've noticed better performance too.
Thank you,
Dominik
|
|
Okay, I finally got a chance to try the new build from master. I don't notice much speed difference; on my system, it looks about the same. This time when I tried it I had enough free memory to load it all on start. I see 112 GB virtual mem and 0.97 GB RSS. That's still huge, though it's about a 20 % improvement. I didn't poke around much, but I added two new entries using JabFox, which is think is a pretty good use-case test. Memory usage after this is 1 GB, so about 20MB increase. That also seems like a crazy amount, but I don't know. This is still much larger than I would expect. I don't know how to estimate the expected memory use for this application, other than by comparing it to other things, like Zotero, and that seems to show that JabRef is much less efficient. Only the developers could say if this is actually the expected memory usage or not, although even that may be difficult. Anyway, if there is something I can do by way of testing, I'm happy to do that. |
76b4268 Style for Acta Physica Sinica (物理学报) (#6009) 604de6f MLA Publication Date update (#5927) 41ce2d4 Update netherlands-journal-of-geosciences-geologie-en-mijnbouw.csl (#6027) ad08f5d Adds space after title writing in ABNT style file (#6031) 0b5c1c2 Create CRCAO Light (#5935) 2f42b8c Create annals-of-laboratory-medicine.csl (#5959) 80a9506 Create annual-review-of-linguistics (#5992) 0fb9c40 Update the-journal-of-egyptian-archaeology.csl (#6028) 5ff53b1 Update guide-des-citations-references-et-abreviations-juridiques.csl (#6002) c1c0212 Update society-for-american-archaeology.csl (#5919) 129ef3c Update administrative-science-quarterly.csl (#5991) 8bc22bd Create multilingual-matters.csl (#5955) aca84fd Create expert-reviews-in-molecular-medicine.csl (#5961) 3c4ddc0 Create ethnobiology-letters.csl (#5986) c301e04 Update heidelberg-university-faculty-of-medicine.csl (#5932) a8c4396 Update tyndale-bulletin.csl (#5949) c18cbcf Bluebook hotfix d950b2b Create incontext-studies-in-translation-and-interculturalism.csl (#5907) 7cfc106 Bump nokogiri from 1.13.2 to 1.13.4 (#6016) 0baa43a Liebert update (#6026) 41ca2b3 Bluebook Add space for ibid (#6025) 6707a37 Update american-journal-of-botany.csl (#5954) 926fad5 Update boletin-de-pediatria.csl (#6024) git-subtree-dir: buildres/csl/csl-styles git-subtree-split: 76b4268
76b4268 Style for Acta Physica Sinica (物理学报) (JabRef#6009) 604de6f MLA Publication Date update (JabRef#5927) 41ce2d4 Update netherlands-journal-of-geosciences-geologie-en-mijnbouw.csl (JabRef#6027) ad08f5d Adds space after title writing in ABNT style file (JabRef#6031) 0b5c1c2 Create CRCAO Light (JabRef#5935) 2f42b8c Create annals-of-laboratory-medicine.csl (JabRef#5959) 80a9506 Create annual-review-of-linguistics (JabRef#5992) 0fb9c40 Update the-journal-of-egyptian-archaeology.csl (JabRef#6028) 5ff53b1 Update guide-des-citations-references-et-abreviations-juridiques.csl (JabRef#6002) c1c0212 Update society-for-american-archaeology.csl (JabRef#5919) 129ef3c Update administrative-science-quarterly.csl (JabRef#5991) 8bc22bd Create multilingual-matters.csl (JabRef#5955) aca84fd Create expert-reviews-in-molecular-medicine.csl (JabRef#5961) 3c4ddc0 Create ethnobiology-letters.csl (JabRef#5986) c301e04 Update heidelberg-university-faculty-of-medicine.csl (JabRef#5932) a8c4396 Update tyndale-bulletin.csl (JabRef#5949) c18cbcf Bluebook hotfix d950b2b Create incontext-studies-in-translation-and-interculturalism.csl (JabRef#5907) 7cfc106 Bump nokogiri from 1.13.2 to 1.13.4 (JabRef#6016) 0baa43a Liebert update (JabRef#6026) 41ca2b3 Bluebook Add space for ibid (JabRef#6025) 6707a37 Update american-journal-of-botany.csl (JabRef#5954) 926fad5 Update boletin-de-pediatria.csl (JabRef#6024) git-subtree-dir: buildres/csl/csl-styles git-subtree-split: 76b4268
JabRef version jabref-5.0.30388-1.x86_64 on
Steps to reproduce the behavior:
What happens is that JabRef Virtual Memory size is 111 GB! And just interacting with the GUI causes a lot of memory to page out, causing some programs to crash. I was running vmstat at the time and it crashed while 2 GB of swap got used in about 30 seconds. Mouse starts jumping and then sound will lock up. Then, the whole computer locked up for about another 30 seconds.
This has happened every time I use JabRef 5 beta packages.
System is Fedora 31, Xorg.
The text was updated successfully, but these errors were encountered: