Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[instagram] fix highlights extraction #2197

Merged
merged 2 commits into from
Jan 23, 2022
Merged

[instagram] fix highlights extraction #2197

merged 2 commits into from
Jan 23, 2022

Conversation

vsyx
Copy link
Contributor

@vsyx vsyx commented Jan 16, 2022

Currently the highlight extractor fails when there are more than 30 reel_ids supplied to the "/v1/feed/reels_media/" endpoint.

For example, fetching highlights from "https://www.instagram.com/f1/" (87 highlights at the time I'm writing this)

gallery-dl -v --no-download -o include="highlights" "https://www.instagram.com/f1/"

will respond with status code 400.

Initially I tried splitting reel_ids into chunks of length 30, however, I quickly found that I consistently get 5xx responses when the total number of reel_ids is above 30 (I waited an entire week just to make sure that I wasn't being rate limited). When the total number of reel_ids is <= 30 (example account "https://www.instagram.com/bemytravelmuse/", 27 highlights at the time I'm writing this), you can fetch everything with a single request, however, in my case 5xx responses can still be frequently observed.

I haven't tested everything thoroughly, but it looks like chunks of 15 seem to work most of the time. That being said, it might be a good idea to start at 30 and just reduce the chunk size (e.g. halve it) until you stop getting 5xx responses.

- 'yield' individual reels instead of collecting them in a list
  and returning them all at once
- reduce 'chunk_size' to an even saver value
  (instagram.com also uses 5)
@mikf mikf merged commit 3f2b633 into mikf:master Jan 23, 2022
@mikf
Copy link
Owner

mikf commented Jan 23, 2022

Thank you.

I reduced the chunk_size even further to a save 5 (same as on instagram itself) since I immediately got a 560 error when testing your solution.
I've also changed it so it doesn't have to wait for all highlight reels to load before starting to download the first file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants