Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

This API is useless #153

Closed
ghost opened this issue Oct 31, 2018 · 5 comments
Closed

This API is useless #153

ghost opened this issue Oct 31, 2018 · 5 comments

Comments

@ghost
Copy link

ghost commented Oct 31, 2018

This is kind of a rant, but what exactly is the point of the deviantart API?

Pretty much every single API out there allows one to do at least a request every couple of seconds constantly. The deviantart API instead regularly loses its mind if anyone dares requesting more than once every what, five seconds? Six?

What is the point of going through the API if I can just use BeautifulSoup or Selenium while logged out and scrape the site multiple times per second without fear of getting banned? Because that's what the API invites you to do, given that the only one who would have it worse in that case is the site.

Yeah, yeah, there's always IP banning, but there's pretty easy workarounds for that.

By comparison, to get a list of a large (~700 posts) gallery's full-res images, it took me 30 minutes with Selenium, and 14 (FOURTEEN!) hours with the official deviantart API, starting with one request every three seconds plus an additional two seconds per pagination request and implementing the exponential wait (first 2, then 4, then 8 seconds etc) as suggested in your docs, because obviously three seconds wasn't enough for the site.

APIs are supposed to be faster because one doesn't have to load all the data that comes with a normal browser's request. If you neuter it to a point where it's quicker for a user to grab the UUIDs by manually inspecting each page's source and doing CTRL+C and pasting it to a txt file, then it's absolutely pointless.

Speaking of which, why in the world would the UUID of a post only be obtainable by scraping the page with a HTML request first? Seriously, why would I even switch to the API and not just grab all the info i need from the HTML page that I had to get, at that point? (this has been discussed before in issue #100 almost two years ago, it's ridiculous that it's still a problem)

I just can't wrap my head around the thought process of whoever designed this API. Not only are you limiting anyone wishing to use it, but your reason to do so (easier load on the server) is immediately invalidated by the fact that anyone who wishes to use it HAS to do HTML scraping in concurrence. You're forcing anyone who wants to use that part of the API to do twice as many requests, and pull the entire html every time.

I really don't want to sound so confrontational, but I've tried to use this API for months, and it's just immensely frustrating.

@ghost
Copy link
Author

ghost commented Nov 4, 2018

Another astounding piece of engineering by the deviantArt team: client credentials is the only method usable without web scraping among their entire api, and it does not grant access to user-specific functions such as the feed. Which means that if you want to have access to a "functional" api you have to scrape the site and simulate a HTML login, because obviously unless you're logged in on a "real" browser their redirect url for the normal Authorization grant won't work - it will just redirect you to the main page.

Again, what's the point of this api?

@rautamiekka

This comment was marked as abuse.

@Petschko
Copy link

Petschko commented Feb 7, 2019

Yeah that API does not make sense, it seems more easy to use a custom build parser instead of their API... Its super sad but just look at their new site (eclipse) they really want to lose worldwide page-rank =/

@muteor
Copy link
Collaborator

muteor commented Oct 15, 2019

Main reason for harsh limits on un-approved clients is because of spammers, this is why we can't have nice things :)

If anyone is having trouble with limits we can look at individual cases, but would need to know more about your planned usage. Feel free to send us a note.

As for API development, currently its not being worked on, we may get back to it once we start updating mobile apps etc but definitely no ETA on this.

Sorry for the frustrations, as with life nothing is perfect and we have limited time to address things.

@muteor muteor closed this as completed Oct 15, 2019
@rautamiekka

This comment was marked as abuse.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants