Deviantart "Too many requests"

[petrarislin]

Trying to use newest gallery-dl under win10 to retrieve photos from a Deviantart account that has been deleted. One can't directly access the account, but the photos are still there with the account name struckthrough. By using search for the account name then "Deviations" tab then "Most recent" option tab (Example: https://www.deviantart.com/search/deviations?q=jamajurabaev&order=most-recent), the still-accessible photos from that account are all shown (and some other detritus like comments made by that author), and can be downloaded manually just like normal. But trying gallery-dl almost immediately shows "[deviantart][warning] API responded with 429 Too Many Requests. Using 1s delay", then 2s, etc. It also doesn't seem be applying the filtering implicit in the path name, instead just retrieving some unrelated photos.

I'd think it's just that the Deviantart extractor isn't built for the search function, but I'm pretty sure it succeeded previously.

Per the returned text, I did try acquiring new oauth credentials, but no change. But I'm not at all sure I put them in the right place, so I'd appreciate explicit instructions for where to put them. I don't otherwise use any conf files. The only outside file gallery-dl creates is %appdata%\gallery-dl\cache.sqlite3, no created conf files I can find.

[rautamiekka]

no created conf files I can find.

There's the problem. https://github.com/mikf/gallery-dl/blob/master/docs/configuration.rst#extractordeviantartclient-id--client-secret

[petrarislin]

Yes, I had read that. I had tried putting gallery-dl.conf (from unzipping the downloadable gallery-dl.exe) in %appdata%\gallery-dl, and replaced the client-id and client-secret entries with the ones which deviantart created, but it didn't seem to make a difference.

Should gallery-dl be creating a conf file somewhere I should use? Or are there more explicit instructions on where to create the gallery-dl.conf? TY.

[mikf]

gallery-dl does not create a config file by itself, but it will automatically read any file at the locations listed here.

I'd recommend you copy docs/gallery-dl.conf, move it into %APPDATA%\gallery-dl next to cache.sqlite3, and rename it to config.json. You can also put it anywhere and use -c/--config to load it.

The OAuth client_id and client_secret values go here:

gallery-dl/docs/gallery-dl.conf

Lines 72 to 73 in a12ce2b

	"client-id": null,
	"client-secret": null,

(don't forget to put them in double quotes)

---

To download from the deleted account, you can just use https://www.deviantart.com/jamajurabaev as URL, although that will only go through all folders and might miss a few deviations. You'd have to comment out a few lines of code to get it to use the /gallery/all endpoint and truly fetch everything.

gallery-dl/gallery_dl/extractor/deviantart.py

Lines 75 to 82 in a12ce2b

	if self.user:
	profile = self.api.user_profile(self.user)
	self.group = not profile
	if self.group:
	self.subcategory = "group-" + self.subcategory
	self.user = self.user.lower()
	else:
	self.user = profile["user"]["username"]

[mikf]

With commit 14717f3, it is now possible to disable the "is this a group" check and download everything without being limited by going folder by folder:

$ gallery-dl -o group=false https://www.deviantart.com/jamajurabaev
/tmp/deviantart/jamajurabaev/deviantart… cub diaper fursona :3 jakub mateusz.png
/tmp/deviantart/jamajurabaev/deviantart_535931328_Lost submarine 5.jpg
/tmp/deviantart/jamajurabaev/deviantart_535930138_Lost submarine 1.jpg
...

[petrarislin]

TY @mikf for the rapid responses! It worked. I was surprised to see gallery-dl find more files than a manual search did for some accounts, and indeed it found the pre-existing folders which search doesn't.

Looks like the main problem was that I hadn't put double quotes around the string part of the client-id and client-secret; since the original value of null wasn't in quotes, I didn't realize the new values needed quotes. I suggest you mention that in the readme.

Also, it doesn't particularly seem to matter if the conf file in that location is named config.json or gallery-dl.conf, right?

I now see what you mean about going folder by folder. In deviantart, the gallery folders consist of "All," "Featured," and folders the owner created. It seems indeed a few files may be in "All" that aren't in any other folder. So I'd like to try the group=false option, but I don't know how to recompile it after editing deviantart.py. Should I just wait for you to push another prebuilt for windows, or are there instructions?

TY again.

[mikf]

I didn't realize the new values needed quotes. I suggest you mention that in the readme.

There is an example in docs/gallery-dl-example.conf that uses quotes

gallery-dl/docs/gallery-dl-example.conf

Lines 112 to 115 in 560f7b4

	"#": "use custom API credentials to avoid 429 errors",
	"client-id": "98765",
	"client-secret": "0123456789abcdef0123456789abcdef",
	"refresh-token": "0123456789abcdef0123456789abcdef01234567",

and I don't think explaining JSON types in this repo is necessary when there are hundreds of other resources that deal with this topic ...

Also, it doesn't particularly seem to matter if the conf file in that location is named config.json or gallery-dl.conf, right?

It does. gallery-dl only checks the 3 listed locations, which does not include %APPDATA%\gallery-dl\gallery-dl.conf. For standalone executable files it also checks for a gallery-dl.conf in the same directory as the .exe file. Maybe that's what you mean?

the gallery folders consist of "All," "Featured," and folders the owner created

As it turns out, "All" is not a real folder. Listing all folders of a user returns everything except "All" and you need to use a different API endpoint to access it.

So I'd like to try the group=false option, but I don't know how to recompile it after editing deviantart.py. Should I just wait for you to push another prebuilt for windows, or are there instructions?

There is a GitHub action that auto-builds executable files every time I do a git push:
https://github.com/mikf/gallery-dl/actions/workflows/executables.yml
Get a file from there, for example this one, and replace your current gallery-dl.exe with it.

(edit: this is mentioned here in the README, by the way)

[petrarislin]

TY again. I tried the newer version. Without the -o group=false option, it found still more items than the prior version, and I like that it kept the folders. But with the group=false option, it not only didn't have the folders, but it found fewer items than the older version had, actually finding only about the number one could find by searching manually using Deviations tab then the Most Recent option. So I'll probably continue using the newer version, without the group=false option. TY!