Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add --download-timeout and --stop-on-failure args #177

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

ILogOutOnTheToilet
Copy link

@ILogOutOnTheToilet ILogOutOnTheToilet commented Jan 8, 2023

add --download-timeout and --stop-on-failure args

I've been having connection timed out issues after some errors while downloading, causing me to be blocked for hours from kemono.party. I don't know 100% what is causing the issue, but this is my best guess and this commit seems to fix the issue:

Due to kemono.party server issues, some images may show on a post as downloadable but throw an error when you try to download them (I have verified this sometimes happen on the browser as well when I clicked on the attachments and I get some server error). And as the code is written now, the program immediately tries to download the next image right after a failed attempt, which causes a rapid succession of GET requests as each download attempt fails. Currently there is no wait time between these downloads, since the program just returns from the download attempt to download the next file. An example of the return statement in the code during a request error:

return

If a post has many images, these instant attempts submit GET requests as fast as your computer can execute for-loops, which is bad and seems to be triggering some rate-limit or DDOS protection that blocks the client for several hours from accessing kemono.party servers.

I added --download-timeout parameter. I tested the change and I can verify that I am no longer being blocked from kemono.party after adding a timeout between downloads of attachments or inline images, although I cannot verify if it's because I just got lucky. I also made some other minor fixes so that the program works properly. The usage example below waits 4 seconds between each attachment and inline download, and on top of that, 5 seconds between each post.

python kemono-dl.py --cookies "kemono.party_cookies.txt" --links "https://kemono.party/patreon/user/<user id>" --post-timeout 5 --ratelimit-sleep 200 --retry 0 --download-timeout 4 --stop-on-failure

I also added a --stop-on-failure parameter to stop downloading completely in case of any request errors. This is useful in the case you don't want to wait and use --download-timeout but still want to avoid risking being blocked by kemono.party for a few hours due to server errors and you want to see the error and then try downloading again later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant