Fix rare ExHentai duplicated metadata bug

[pink-red]

I'm using gallery-dl as a library and I've created a custom job for metadata extraction, similar to DataJob. But I was getting identical metadata for different pages on ExHentai. Digging a bit deeper, here's what I've found.

ExhentaiGalleryExtractor.items yields the same object for each image in a gallery. This causes problems when the object is not immediately used.

To reproduce:

gallery-dl --dump-json https://e-hentai.org/g/2346908/ab876a4073/ -o output.private=true

-o output.private=true causes image kwdict to be passed through util.identity in DataJob.handle_url:

gallery-dl/gallery_dl/job.py

Line 700 in 560f7b4

self.filter = util.identity if private else util.filter_dict

gallery-dl/gallery_dl/job.py

Lines 734 to 735 in 560f7b4

	def handle_url(self, url, kwdict):
	self.data.append((Message.Url, url, self.filter(kwdict)))

Since util.identity doesn't have a side-effect of creating a new dictionary like util.filter_dict, and since DataJob doesn't use the objects immediately, this triggers the bug.

[mikf]

It is not just the exhentai extractor that reuses the same dict over and over during its data extraction process. This is more of a general trend among all extractor modules, and only "fixing" exhentai does therefore not make too much sense.

This whole mess will eventually be fixed and every file returned by an extractor will be its own independent object together with a change of how extractors return there results, but that will most likely be a lot of work and is planned for v2.0.

In your case, I'd just replace util.identity with dict.copy for DataJob's filter. Does essentially the same and returns a copy.

[pink-red]

In your case, I'd just replace util.identity with dict.copy for DataJob's filter. Does essentially the same and returns a copy.

Ah, indeed, that would be much simpler. Thank you!