It's possible to avoid some pointless deviantart avatar requests

[Twi-Hard]

If ?https://a.deviantart.net/avatars/default.gif is appended to an avatar url then it will have no avatar. I've tested many manually and every attempt shown in the logs 404. I already have 5000 lines in my logs that match ?https://a.deviantart.net/avatars/default.gif and I just started. It should save a lot of time and requests if gallery-dl didn't try to download these links. It's not super important I guess but I thought I should mention it. :)

2024-03-03 16:46:47 debug: https://a.deviantart.net:443 "GET /avatars-big/s/p/spunmetal.gif?https://a.deviantart.net/avatars/default.gif HTTP/1.1" 404 4822
2024-03-03 16:46:47 Warning: '404 Not Found' for 'https://a.deviantart.net/avatars-big/s/p/spunmetal.gif?https://a.deviantart.net/avatars/default.gif'
2024-03-03 16:46:48 debug: https://a.deviantart.net:443 "GET /avatars/s/p/spunmetal.gif?https://a.deviantart.net/avatars/default.gif HTTP/1.1" 404 4822
2024-03-03 16:46:48 Warning: '404 Not Found' for 'https://a.deviantart.net/avatars/s/p/spunmetal.gif?https://a.deviantart.net/avatars/default.gif'
2024-03-03 16:46:49 debug: https://a.deviantart.net:443 "GET /avatars-original/t/h/thatantifurr.png?https://a.deviantart.net/avatars/default.gif HTTP/1.1" 404 4822
2024-03-03 16:46:49 Warning: '404 Not Found' for 'https://a.deviantart.net/avatars-original/t/h/thatantifurr.png?https://a.deviantart.net/avatars/default.gif'
2024-03-03 16:46:50 debug: https://a.deviantart.net:443 "GET /avatars-big/t/h/thatantifurr.png?https://a.deviantart.net/avatars/default.gif HTTP/1.1" 404 4822
2024-03-03 16:46:50 Warning: '404 Not Found' for 'https://a.deviantart.net/avatars-big/t/h/thatantifurr.png?https://a.deviantart.net/avatars/default.gif'
2024-03-03 16:46:50 debug: https://a.deviantart.net:443 "GET /avatars/t/h/thatantifurr.png?https://a.deviantart.net/avatars/default.gif HTTP/1.1" 404 4822
2024-03-03 16:46:50 Warning: '404 Not Found' for 'https://a.deviantart.net/avatars/t/h/thatantifurr.png?https://a.deviantart.net/avatars/default.gif'
2024-03-03 16:46:50 debug: https://a.deviantart.net:443 "GET /avatars-original/t/h/thatantifurr.jpg?https://a.deviantart.net/avatars/default.gif HTTP/1.1" 404 4822
2024-03-03 16:46:50 Warning: '404 Not Found' for 'https://a.deviantart.net/avatars-original/t/h/thatantifurr.jpg?https://a.deviantart.net/avatars/default.gif'
2024-03-03 16:46:51 debug: https://a.deviantart.net:443 "GET /avatars-big/t/h/thatantifurr.jpg?https://a.deviantart.net/avatars/default.gif HTTP/1.1" 404 4822
2024-03-03 16:46:51 Warning: '404 Not Found' for 'https://a.deviantart.net/avatars-big/t/h/thatantifurr.jpg?https://a.deviantart.net/avatars/default.gif'
2024-03-03 16:46:51 debug: https://a.deviantart.net:443 "GET /avatars/t/h/thatantifurr.jpg?https://a.deviantart.net/avatars/default.gif HTTP/1.1" 404 4822
2024-03-03 16:46:51 Warning: '404 Not Found' for 'https://a.deviantart.net/avatars/t/h/thatantifurr.jpg?https://a.deviantart.net/avatars/default.gif'
2024-03-03 16:46:52 debug: https://a.deviantart.net:443 "GET /avatars-original/t/h/thatantifurr.gif?https://a.deviantart.net/avatars/default.gif HTTP/1.1" 404 4822
2024-03-03 16:46:52 Warning: '404 Not Found' for 'https://a.deviantart.net/avatars-original/t/h/thatantifurr.gif?https://a.deviantart.net/avatars/default.gif'
2024-03-03 16:46:52 debug: https://a.deviantart.net:443 "GET /avatars-big/t/h/thatantifurr.gif?https://a.deviantart.net/avatars/default.gif HTTP/1.1" 404 4822
2024-03-03 16:46:52 Warning: '404 Not Found' for 'https://a.deviantart.net/avatars-big/t/h/thatantifurr.gif?https://a.deviantart.net/avatars/default.gif'
2024-03-03 16:46:52 debug: https://a.deviantart.net:443 "GET /avatars/t/h/thatantifurr.gif?https://a.deviantart.net/avatars/default.gif HTTP/1.1" 404 4822
2024-03-03 16:46:52 Warning: '404 Not Found' for 'https://a.deviantart.net/avatars/t/h/thatantifurr.gif?https://a.deviantart.net/avatars/default.gif'

Edit: When I did testing with avatars in the past, anything after the ? didn't seem to matter at all so maybe the ? and everything after it should be removed?

[mikf]

The part after the ? is supposed to just be the avatar index number, e.g. 4 for https://a.deviantart.net/avatars-big/s/h/shimoda7.jpg?4, but the current code seems to put the entire URL there.

Some questions:

• should default avatars get ignored or do you want to download them?
• should users with default avatars still go through all possible avatar formats?
  (https://a.deviantart.net/avatars/t/h/thatantifurr.gif etc, even though this user has a default avatar)

[Twi-Hard]

According to the logs, none of the formats/sizes succeded when ?https://a.deviantart.net/avatars/default.gif was in the URL. The default avatar shouldn't be downloaded for sure. There's only 1 and it's easy to get manually. The default avatar can be in different sizes for different users for some reason so it would still take potentially multiple requests per user that didn't upload a custom pic (which is a huge percentage of them).

[Twi-Hard]

When I did my testing in the past I downloaded every format/size combination without the ? and the stuff after it by using the list of usernames to recreate the avatar urls with a script. For some reason it was downloading default avatars using those recreated links sometimes (not the regular default avatar url we're seeing after the ?). So I'm thinking it's possible to run into the default avatar on some users anyways? By that I mean the normal avatar urls can contain the default avatar without us knowing it's not a custom one potentially? If so I'm not sure how gallery-dl could detect that (unless if that only happened on accounts where gallery-dl would have put ?https://a.deviantart.net/avatars/default.gif in the URL so the default avatars with normal avatar urls would be skipped anyways once not attempting to download avatars if ?https://a.deviantart.net/avatars/default.gif is in the URL is implemented).

Hopefully I didn't say that too confusingly

Edit: this post gives context to what I meant by the default avatars can be different sizes for different users.

[mikf]

The bug with putting the entire URL after ? is fixed (0cbc910)
and default avatars now get ignored (a767832).

Let me know if you have any other problems in this regard or close this issue otherwise.