Encountering connection timed out errors from Kemono after running this

[ILogOutOnTheToilet]

Description

Used this tool again just today and it caused a bunch of 403 errors at first and failed to download file errors. Now I tried again and it caused connection timed out errors.

This is the error when I tried to download again:

requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='kemono.party', port=443): Max retries exceeded with url: /api/creators/ (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x00000239AFDA41C0>, 'Connection to kemono.party timed out. (connect timeout=300)'))
ERROR:No creator information was retrieved. | exiting

Same error "connection timed out" error when visiting the website in any browser. I have verified that the website isn't down by checking it in one of those "is a website down?" websites.

Is this due to some new rate limiting? If so, default timeout and retry values should be reconfigured since otherwise downloading a bunch of any user's stuff will cause you to be locked out from Kemono servers for I don't know how long. Maybe there is a dynamic way to do this? But I am not familiar with Kemono server APIs.

Service, User ID, Post ID

Doesn't matter what Service, User ID, or Post ID since connection is timed out trying to visit even the website home page.

[L4cache]

are you using shared proxies? i didn't encounter this issue.

[ILogOutOnTheToilet]

are you using shared proxies? i didn't encounter this issue.

I don't think so. Maybe my ISP is doing it? If so I never heard of such a thing. Have you downloaded an entire user's post history, that's what caused it for me. The first several posts were fine, but after a while, the error occurred.

[L4cache]

are you using shared proxies? i didn't encounter this issue.

I don't think so. Maybe my ISP is doing it? If so I never heard of such a thing. Have you downloaded an entire user's post history, that's what caused it for me. The first several posts were fine, but after a while, the error occurred.

I've downloaded hundreds of post and never have this issue...

[ILogOutOnTheToilet]

@L4cache I got it again, it started with a bunch of 403 errors. I think it might be downloading images of a single post too fast? I added timeout between posts of 5 seconds, but the error started right after the first post download. There isn't a parameter option to add wait time between image downloads of a post. I added the code to see if it fixes it, but it seems to lock you out of the website for a few hours every time it happens, so it will take a while to verify.

[ILogOutOnTheToilet]

I don't know 100% what is causing the issue, but this is my best guess:

Due to kemono.party server issues, some images may show on a post as downloadable but throw an error when you try to download them (I have verified this sometimes happen on the browser as well when I clicked on the attachments and I get some server error). And as the code is written now, the program immediately tries to download the next image right after a failed attempt, which causes a rapid succession of GET requests as each download attempt fails. Currently there is no wait time between these downloads, since the program just returns from the download attempt to download the next file. An example of the return statement in the code during a request error:

kemono-dl/src/main.py

Line 461 in 5bfab2e

return

If a post has many images, these instant attempts submit GET requests as fast as your computer can execute for-loops, which is bad and seems to be triggering some rate-limit or DDOS protection that blocks the client for several hours from accessing kemono.party servers.

I added --download-timeout parameter. I tested the change and I can verify that I am no longer being blocked from kemono.party after adding a timeout between downloads of attachments or inline images, although I cannot verify if it's because I just got lucky. I also made some other minor fixes so that the program works properly. The usage example below waits 4 seconds between each attachment and inline download, and on top of that, 5 seconds between each post.

python kemono-dl.py --cookies "kemono.party_cookies.txt" --links "https://kemono.party/patreon/user/<user id>" --post-timeout 5 --ratelimit-sleep 200 --retry 0 --download-timeout 4 --stop-on-failure

I also added a --stop-on-failure parameter to stop downloading completely in case of any request errors. This is useful in the case you don't want to wait and use --download-timeout but still want to avoid risking being blocked by kemono.party for a few hours due to server errors and you want to see the error and then try downloading again later. I haven't thoroughly tested this since it requires consistent reproduction of server errors from kemono.party, but I have tested that it doesn't interfere with normal download process and that it stops program execution when it encounters an exception when the parameter is passed.

Here is the PR:
#177