Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error - Imgur [429 Unknown Error] (Rate-Limit) #1386

Closed
sourmilk01 opened this issue Mar 17, 2021 · 2 comments
Closed

Error - Imgur [429 Unknown Error] (Rate-Limit) #1386

sourmilk01 opened this issue Mar 17, 2021 · 2 comments

Comments

@sourmilk01
Copy link

When scraping reddit, I've been getting "429 Unknown Error" for imgur-hosted images. From what I've read, Error 429 deals with rate limits for the imgur API.

Is there any way to better manage the requests from imgur? Would setting up authentication for imgur work?
@sourmilk01
Copy link
Author

Might be on imgur's side, I was trying to investigate further but https://api.imgur.com/ returns a 403 code: "Imgur is temporarily over capacity. Please try again later".

mikf added a commit that referenced this issue Mar 18, 2021
@mikf
[imgur] fix/improve rate limit handling (#1386)
0b55f5a 
- also wait-and-retry on 429 status codes
- use infinite loop instead of recursive calls
- 'extractor.sleep()' -> 'extractor.wait()'
@mikf
Copy link
Owner

mikf commented Mar 18, 2021

Might be on imgur's side

That's probably the case. gallery-dl uses imgur's own API key (the one used when browsing the website in a browser), which appears to not be rate limited at all ... at least under normal circumstances. API requests with it have X-Ratelimit-... headers, but the limit never goes down.

0b55f5a improves the rate limit and "over capacity" handling a bit and should just wait a bit in case this happens again.

@mikf mikf closed this as completed Mar 18, 2021
@sourmilk01 sourmilk01 mentioned this issue Mar 26, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mikf @sourmilk01