Best way to keep Gelbooru favorites up-to-date?

[Hawker2]

I've been attempting to keep my favorites up-to-date using gallery-dl, and recently noticed that I was only getting recently posted images on Gelbooru; if I favorited an older image, it wouldn't be downloaded. Looking at the verbose output, this seems to be because the Favorites API is returning the images in reverse order of image id, not the time added as a favorite (discussed on issue #3704 ). Is there any way to properly get recently-added favorites without scanning the entire list? Archive file doesn't seem to help, as that seems to deal with already-existing files, and doesn't stop gallery-dl from making network requests (two per favorite - which adds up when you have thousands of favorites).

Here's the command line:
gallery-dl --ignore-config -c "config-gelbooru-favorites-test.json" "https://gelbooru.com/index.php?page=favorites&s=view&id=REDACTED"

Here's config-gelbooru-favorites-test.json:

{
	"extractor":
	{
		"skip": true,
		"path-restrict": "windows",
		"path-strip": "windows",
		"sleep": 10,
		"sleep-request": 0.25,
		"retries": 4,
		"user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0",
		
		"gelbooru": {
			"base-directory": "/REDACTED/favorites/",
			"directory": [],
			"archive": "/REDACTED/favorites/archive.db",
			"filename": "{id}.{extension}",
			"metadata": true,
			"tags": true,
			"skip": "abort:10",
			"user-id": "REDACTED",
			"api-key": "REDACTED",
			"postprocessors": [
				{
					"name": "metadata",
					"mode": "json",
					"directory": ["metadata"],
					"filename": "{id}.json",
					"skip": true,
					"event": "post"
				}
			]
		}
	},
	
	"output":
	{
		"skip": false
	}
}

I've also tried supplying cookies to see if being "logged in" helps, and tried supplying username/password keys (even though documentation indicates these aren't used for Gelbooru, I figured it couldn't hurt). No change; verbose output still shows IDs being returned in reverse-id order.

Other relevant detail:

[gallery-dl][debug] Version 1.26.1
[gallery-dl][debug] Python 3.11.5 - Linux-3.10.108-x86_64-with-glibc2.36
[gallery-dl][debug] requests 2.31.0 - urllib3 2.0.4
...
[gelbooru][debug] Using GelbooruFavoriteExtractor for 'https://gelbooru.com/index.php?page=favorites&s=view&id=REDACTED'

[mikf]

I did a few test runs and, at least for me, it returns favorites in the same order as displayed by Gelbooru, which is descending favorite date order, I think.

$ gallery-dl --filter "print(id)" "https://gelbooru.com/index.php?page=favorites&s=view&id=279415"
1761698
4968977
2152973

this seems to be because the Favorites API is returning the images in reverse order of image id,

It's returning favorites in ascending order of fav date, and I spend far longer than I want to admit to have gallery-dl return them in reverse / descending order. (dcb8af6)

[Hawker2]

Now that is odd. This is certainly the way I recall it working previously, yet it's not for me:

~ $ gallery-dl --ignore-config "https://gelbooru.com/index.php?page=favorites&s=view&id=REDACTED" -s -v
[gallery-dl][debug] Version 1.26.1
[gallery-dl][debug] Python 3.11.5 - Linux-3.10.108-x86_64-with-glibc2.36
[gallery-dl][debug] requests 2.31.0 - urllib3 2.0.4
[gallery-dl][debug] Configuration Files []
[gallery-dl][debug] Starting SimulationJob for 'https://gelbooru.com/index.php?page=favorites&s=view&id=REDACTED'
[gelbooru][debug] Using GelbooruFavoriteExtractor for 'https://gelbooru.com/index.php?page=favorites&s=view&id=REDACTED'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): gelbooru.com:443
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&s=favorite&id=REDACTED&limit=1 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&s=favorite&id=REDACTED&limit=100&pid=220 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9196504&s=post HTTP/1.1" 200 None
gelbooru_9196504_4c04def15383133c3b931478b6f7ec82.jpg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9196428&s=post HTTP/1.1" 200 None
gelbooru_9196428_a9e274014dddf4d0c23674d4be5b3dd8.jpg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9196407&s=post HTTP/1.1" 200 None
gelbooru_9196407_17dffcc0c9efb6089400d91d53309f59.png
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9196084&s=post HTTP/1.1" 200 None
gelbooru_9196084_141cff68e4db88971bf5d3d8e34e4d4b.png
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9196040&s=post HTTP/1.1" 200 None
gelbooru_9196040_a15456ac1af7c2373dd8c5c17003862f.jpg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9195851&s=post HTTP/1.1" 200 None
gelbooru_9195851_f770e82b241ff7d1f17d0b7d5a9a9bdd.png
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9195838&s=post HTTP/1.1" 200 None
gelbooru_9195838_bc8ca0001ac9ea234e448dcba56963f0.jpg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9195600&s=post HTTP/1.1" 200 None
gelbooru_9195600_ec4a89540ec927d46b82cad5dc22cc49.jpg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9195574&s=post HTTP/1.1" 200 None
gelbooru_9195574_aa8f486ddab78bc57c3e0bef4cff441e.jpg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9195565&s=post HTTP/1.1" 200 None
gelbooru_9195565_787af75f921df6e9196a56233b4bff24.jpg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9195482&s=post HTTP/1.1" 200 None
gelbooru_9195482_4cf7d0d5f54fc11336fa8358a769a477.jpg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9195359&s=post HTTP/1.1" 200 None
gelbooru_9195359_8b859cb743d5c44a1e2b031925f247db.jpg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9195182&s=post HTTP/1.1" 200 None
gelbooru_9195182_b8b895cd09f9df407117fefd49a21c7f.jpg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9194846&s=post HTTP/1.1" 200 None
gelbooru_9194846_02ac99a464542c3c8c248a61e0840352.jpg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9194775&s=post HTTP/1.1" 200 None
gelbooru_9194775_351ac405df5eae5426bd05602c2a23bd.png
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9194764&s=post HTTP/1.1" 200 None
gelbooru_9194764_51e4434607eca5f910c77f74171702f3.jpg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9189012&s=post HTTP/1.1" 200 None
gelbooru_9189012_dea4c67e18e4a380954b9fe7cba6c493.jpg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9182509&s=post HTTP/1.1" 200 None
gelbooru_9182509_54842add989862dd081abb775f69d017.jpg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9182450&s=post HTTP/1.1" 200 None
gelbooru_9182450_6f957a5b219752733cf48f97d477cb9f.jpeg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9181964&s=post HTTP/1.1" 200 None
gelbooru_9181964_9b7fd2f5ceb6a1fb4b7379df29c5f101.jpg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9181100&s=post HTTP/1.1" 200 None
gelbooru_9181100_4eb7404082b057a61fbf03235f192754.jpeg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9180794&s=post HTTP/1.1" 200 None
gelbooru_9180794_ffd6adf85c409d7f02d085687474fbff.jpg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9180781&s=post HTTP/1.1" 200 None
gelbooru_9180781_c077f103df8cc4a4c10fa44682845f02.jpeg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=9180652&s=post HTTP/1.1" 200 None
gelbooru_9180652_5916e6f944bbcd122af56b7180d5ad92.png
^C
KeyboardInterrupt

Looking at the favorites on the Gelbooru website, they're ordered 9189012, 9194764, 9194775, etc, which is not what gallery-dl is getting back from the API. With "--ignore-config" and no extra config specified, I really should be getting default behavior, shouldn't I?

[Hawker2]

Okay, there's something VERY odd going on here, as I see (expected) unsorted behavior with your ID, and (unexpected) sorted behavior with mine!

~ $ gallery-dl --ignore-config "https://gelbooru.com/index.php?page=favorites&s=view&id=279415" -s -v
[gallery-dl][debug] Version 1.26.1
[gallery-dl][debug] Python 3.11.5 - Linux-3.10.108-x86_64-with-glibc2.36
[gallery-dl][debug] requests 2.31.0 - urllib3 2.0.4
[gallery-dl][debug] Configuration Files []
[gallery-dl][debug] Starting SimulationJob for 'https://gelbooru.com/index.php?page=favorites&s=view&id=279415'
[gelbooru][debug] Using GelbooruFavoriteExtractor for 'https://gelbooru.com/index.php?page=favorites&s=view&id=279415'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): gelbooru.com:443
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&s=favorite&id=279415&limit=1 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&s=favorite&id=279415&limit=100&pid=0 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=1761698&s=post HTTP/1.1" 200 None
gelbooru_1761698_6abb21580479c27c5394dc5f9fe4ecb3.jpeg
[urllib3.connectionpool][debug] https://gelbooru.com:443 "GET /index.php?page=dapi&q=index&json=1&id=4968977&s=post HTTP/1.1" 200 None
gelbooru_4968977_06f252f615a85f3f3d1fbb53549eacbf.gif
^C
KeyboardInterrupt

I'm going to follow this with the Gelbooru devs.

[Hawker2]

So the Gelbooru devs disappeared into the night (I suppose I should have expected that given the state of their forums, documentation, FAQs, etc) but at least it's now working as before. But I have to wonder what kind of bug leads to different user ids functioning in completely different ways - that's just madness. Sorry to take up your time with this silliness!

[Hawker2]

This doesn't need to be reopened, but for anyone who runs across this in the future - it seems that this just happens sometimes with Gelbooru. For instance, API is currently returning my favorites sorted by post ID again (rather than date added), but in the last couple days it's even swapped the order... of sorting the wrong column. So now gallerydl gets posts starting with the oldest ones on the site, moving forward to the most recent - regardless of when I might have added them to favorites.

(Don't get me wrong, the Gelbooru devs run an amazing, performant, pretty-much-never-down site. Massive kudos. But if it's not "core mission" (see forums, documentation, API, etc), it really gets the shaft.)

[9696neko]

I seem to have hit this issue as well? When I run my usual command I get immediately finished with my "abort:3" on the same three images that seem to be from the beginning of favourited time. This seems to imply some weird sort ordering is going on.