add --download-timeout and --stop-on-failure args #177
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
add --download-timeout and --stop-on-failure args
I've been having connection timed out issues after some errors while downloading, causing me to be blocked for hours from kemono.party. I don't know 100% what is causing the issue, but this is my best guess and this commit seems to fix the issue:
Due to kemono.party server issues, some images may show on a post as downloadable but throw an error when you try to download them (I have verified this sometimes happen on the browser as well when I clicked on the attachments and I get some server error). And as the code is written now, the program immediately tries to download the next image right after a failed attempt, which causes a rapid succession of GET requests as each download attempt fails. Currently there is no wait time between these downloads, since the program just returns from the download attempt to download the next file. An example of the return statement in the code during a request error:
kemono-dl/src/main.py
Line 461 in 5bfab2e
If a post has many images, these instant attempts submit GET requests as fast as your computer can execute for-loops, which is bad and seems to be triggering some rate-limit or DDOS protection that blocks the client for several hours from accessing kemono.party servers.
I added
--download-timeout
parameter. I tested the change and I can verify that I am no longer being blocked from kemono.party after adding a timeout between downloads of attachments or inline images, although I cannot verify if it's because I just got lucky. I also made some other minor fixes so that the program works properly. The usage example below waits 4 seconds between each attachment and inline download, and on top of that, 5 seconds between each post.python kemono-dl.py --cookies "kemono.party_cookies.txt" --links "https://kemono.party/patreon/user/<user id>" --post-timeout 5 --ratelimit-sleep 200 --retry 0 --download-timeout 4 --stop-on-failure
I also added a
--stop-on-failure
parameter to stop downloading completely in case of any request errors. This is useful in the case you don't want to wait and use--download-timeout
but still want to avoid risking being blocked by kemono.party for a few hours due to server errors and you want to see the error and then try downloading again later.