-
-
Notifications
You must be signed in to change notification settings - Fork 45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG?] very slow in creating library #427
Comments
Hey 👋 Thanks for the detailed report!
Coincidentally, someone else on Discord reported the really slow thumbnail generation. I'm not sure if you're on there, but if you're interested the discussions can be found here. There is already a draft PR up which I'm hoping will help with this.
Can you share additional details about your library structure? You said a single folder, so something like?:
OR
Or something else? I'm curious if perhaps you are being affected by a recently fixed bug in the scanner (not in latest yet): #423
This is an odd scenario. I won't have time to review the logs you provided today, but do you notice the server doing anything when this happens? I'm curious if there is a panic happening somewhere.
If I had to guess, CPU usage is low during a scan but huge during thumbnail generation. A scan doesn't load too much of the files into memory, but to generate thumbnails means we must load a (potentially large) image into memory, convert it to another format, apply transformations (scale, quality, etc). This is another area I'm hopeful the aforementioned PR will help, though. Edit to add: I assume you are also using a network drive for your books ( |
Thanks for the quick reply, I will just experiment around, stump is (not yet) my main comic reader (but will probably be as Komga is probably never going to implement smart filters). As for your remarks:
The first test library was just 40 books in the root folder, my regular library is roughly as follows, one folder per
Nothing that I am aware of, any idea where I could be looking?
I know that this could be a bottleneck (which is why I switched to NFS instead of SMB, the files are served by a Synology NAS) but other comic readers (tried Kavita & Komga) currently do not suffer from any problems. P.S.: memory usage is now up to 5.8GB, this does somehow not seem quite right... Is this due to the very verbose logging? |
This sounds like it would trigger that bug I mentioned. The bug is specifically for media files existing in the root of a library. Since it was just a test, it shouldn't otherwise affect you (unless you have files in the library root elsewhere). It should be fixed as of the last
Nothing specific, I really am curious what the entire scan would output though. No books is odd, but sporadically is even stranger. In general, keywords to look for in logs include
I wouldn't think so, you're mapping to a dir on your host where the logs are written to. I am not sure how long the logs would remain in a buffer somewhere if nobody is attached to it. Would you be open to spinning up a test container if I were to push an image with the changes from #426? I think your setup is a good one for working out some of these issues. |
Absolutely, just tell me what to do. Might take until Monday, though. One thing that I noticed after starting a new library scan with reduced log verbosity (the only real error I found in the log):
|
I appreciate it! No rush on my end. I'll be pushing a Edit to add: it just got pushed https://hub.docker.com/r/aaronleopold/stump/tags There are two new configuration options to try out:
Without more verbose logs it's hard to say would could have triggered a 500. I wouldn't necessarily attribute it to the scanner though, likely a red herring. When you start a scan via HTTP call, it sends an internal message to queue the job. The HTTP request should end by the time the job would start. The only time a 500 would happen is if the message to queue the job failed, but since it shows some info level logs from the scanner I don't think that is what is happening. |
See #426 (comment) for some more context |
I tried the experimental docker image but I guess there is something more afoot here with my setup than just slow thumbnail creation. Somehow, even after two days, the library creation & scanning process was still working and the status just showed I recreated the container after deleting all user data and it is currently working. 2/125 tasks done after 50 minutes of a library with ~700 books/100GB and memory consumption is steadily rising with |
To be clear, when you say you tried the experimental docker image you mean the literal
Yeah that sounds like it was overwhelmed, e.g. no resources available to issue the queries to display misc settings sections so the suspense boundary stays (nothing gets rendered). If you don't have the Is there any way for you to profile what is eating the memory, specifically? E.g. if the process is holding any files open, etc. |
I was using
Sorry, I'm not much of a developer (not at all) and thus cannot really help there. If you have any advice on what you'd need, I can go from there. From what I tried:
edit: stupid me thought that the percentage indicator in the upper right corner was the completion of library building but instead is the reading completion... |
No worries! I appreciate what you provided. Unfortunately nothing really sticks out to me, aside from curiosity around those I was able to get a quickly growing memory usage for pretty much any scenario (host mount, SMB, NFS), getting to 10GB easily within 20 minutes. When cancelling the scan, the memory stabilizes and the OS does not free it. I just killed the container after 30 minutes sitting around 10GB idle. I'm not 100% what the next steps should be TBH, since I only am able to experience this in docker I'm thinking:
|
Okay so I might have found something! I think the root cause of this is an issue in the
It's important to note that the
The results were pretty much the same between the versions. The memory usage definitely did still increase during the duration of the scan, but we're talking magnitudes less than before. I was getting to over 10GB of usage within 20 minutes of scanning the NFS mount, but with the downgraded I think next steps will be to push another I'll also try to find time to create a good reproduction to hand off to the |
@Cantello I'm going to plan on merging the linked PR sometime over the weekend. If you're able to, let me know whether the latest |
I just tried the latest per-experimental container again and yes, it has greatly improved the situation here. The whole library (for one language, 700 files, 100GB) was scanned and thumbnails were created in ~14 minutes. Memory usage is high (1.8GB) but not dramatically so. |
Awesome, thanks for re-testing with the updates! I'll go ahead and close out this issue now that the slowness aspect has been resolved. I'll try and keep an eye on the memory usage in docker, it's odd that running Stump outside docker doesn't have the same issue. If you find the memory worsens, feel free to create a separate issue. Thanks for outlining your experience and pointing the slowness out! You directly helped in figuring out the huge performance regression 🤝 |
Describe the bug
When adding a library, the media entities are sometimes added quickly but thumbnails take forever or are not generated at all, sometimes, the entities are not created at all.
To Reproduce
Expected behavior
Library is created with thumbnails
Screenshots
Click to expand
Desktop (please complete the following information):
Access on Windows, Vivaldi as browser, stump is hosted on docker on debian 12.
Additional context
Log file here (maybe not enough and/or too chatty, was using verbosity 3):
https://logpaste.com/6lqmqRvU
The container stats are also interesting, nearly no CPU usage (1-2%) but huge memory usage (2.5GB after 15 minutes and rising).
The text was updated successfully, but these errors were encountered: