[derpibooru] Inevitable fatal '429 Too Many Requests' for 'https://derpibooru.org/api/v1/json/search/images'

[rautamiekka]

Needed to force myself to report this problem at some point ...

1. Derpibooru sends the error in the title, and gdl quits altogether when it happens. The line is the very last one before returning to the terminal in such case.

2. Dunno since when anymore or why, I think it was somewhere around 1.20.0 the 1st time, but don't quote me on that, I know it happened long before the now current 1.25.7 and the previous 1.25.5-6.

3. When this happens is also random but almost guaranteed the closer to 50 pages of 50 items each I get, and seemingly exponentially more likely since.

4. I remember that back then I immediately checked the FAQ & API docs & the Forums on the site, also the Philomena GitHub, but found absolutely no changes or even remote mentions.

5. Since I've almost 7300 favs that totals 146 pages, I haven't been able to fully download them for about 9 months. It's that bad and long-term.

6. I do use sleep and sleep-request specifically for Derpibooru but even high values only postpone the inevitable the vast majority of the time, so I've limited both to something more reasonable.

7. gdl is installed from the GitHub repo by pip3:
   pip3 install --upgrade 'git+https://github.com/mikf/gallery-dl'

8. Used NSFW link: https://derpibooru.org/search?q=faved_by_id%3A393229

Config: https://paste.gg/p/rautamiekka/3435156f501942d38f8c4846f0de090e

PowerShell output: https://paste.gg/p/rautamiekka/14002e2e2afa46c4ac3b68c56cadec2c

[mikf]

The docs also say nothing about any potential rate limits and there are no x-ratelimit- headers or similar in the response headers.

I guess catching 429 errors and waiting a few minutes might help.

In the meantime, to at least somewhat solve your problem, you could use --range to jump ahead without wasting any API requests. (a1ffa1f)

[rautamiekka]

Ya, the API docs say nothing way or another. Time to ask them for clarification.

[mikf]

It'll now retry 429 errors after waiting for 10min (fceabee)