Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multiple profile images: avoid rate limit for more than 40 users? #35

Open
lazytownfan opened this issue Aug 4, 2022 · 3 comments
Open

Multiple profile images: avoid rate limit for more than 40 users? #35

lazytownfan opened this issue Aug 4, 2022 · 3 comments

Comments

@lazytownfan
Copy link
Contributor

After issues #31 and #32 were fixed, I noticed there is rate limiting if I use $ vsco-scraper -mp vsco-list.txt for downloading multiple profile images with a text file when the list of usernames is longer than 40 (but shorter than 100). (Side note: the command flags have to strictly be placed before the text file name in version 0.7.0.)

After the 40th user has been checked, every single username starting with the 41st will crash.

However, I am able to run $ vsco-scraper -m vsco-list.txt to download multiple gallery images from the same list with more than 40 usernames with no rate limiting issues.

I don't know where I saw or read this, but I think there was some measure to avoid rate limiting implemented when downloading multiple galleries/journals/collections of users IIRC - which is really helpful when you use a text file. Can the same be implemented for downloading multiple profile images?
@intothevoid33
Copy link

I have noticed limiting with the updated script as well.

I've never used the text list, however. I have a bash script that goes one-by-one through the list. I added a 5-second delay between each profile and haven't had any issues since.

If there's no ability to rate-limit between profiles with the current script and text-file, that would be a great addition.

@lazytownfan
Copy link
Contributor Author

Before version 0.7.0, the main component of my Bash alias contained:

$ vsco-scraper -mp vsco-list.txt && vsco-scraper -m vsco-list.txt

before the rate limiting of multiple profile images came up and caused the latter half from executing.

For me (even with a VPN), running the latter half $ vsco-scraper -m vsco-list.txt for VSCO main galleries seems to be free from rate limiting.

@intothevoid33 Do you have any advice on my loop? I tried the following:

$ while read line; do vsco-scraper -p $line && echo -e && sleep 5; done < vsco-list.txt

but after roughly the 40th user, I still get crashes.

However, this std out error messages repeats for every user after the 41st user:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.local/bin/vsco-scraper", line 8, in <module>
    sys.exit(main())
  File "/home/user/.local/lib/python3.10/site-packages/vscoscrape/vscoscrape.py", line 776, in main
    scraper = Scraper(args.username)
  File "/home/user/.local/lib/python3.10/site-packages/vscoscrape/vscoscrape.py", line 30, in __init__
    self.newSiteId()
  File "/home/user/.local/lib/python3.10/site-packages/vscoscrape/vscoscrape.py", line 60, in newSiteId
    self.siteid = res.json()["sites"][0]["id"]
  File "/usr/lib/python3.10/site-packages/requests/models.py", line 917, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] <html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
</body>
</html>
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
<!-- a padding to disable MSIE and Chrome friendly error page -->
: 0

In my previous testing, using the text file would only show me that username-here crashed after the 40th user with $ vsco-scraper -mp vsco-list.txt. However, even after this command crashes from rate limiting, I am able to manually enter $ vsco-scraper -p remaining-name-here for remaining users (almost 40 more) after the 40th one with seemingly no rate limiting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lazytownfan @intothevoid33 and others