Reconstructing DeviantArt gallery/folder structure

[ReneRanzinger]

Since the flat configuration argument seems not to be working for now (see #271) is it possible to add a feature that allows to reconstruct the DeviantArt gallery/folder structure? For example by adding the association of an image with folder(s) in the metadata json file of the image or by generating an individual file containing this association for all images?

[mikf]

There is currently no such option, and flat will be fixed when f837ea9 is released in the next version.

flat=false works by going through the list of all gallery folders, but that has its downsides:

• deviations can be in no folder at all and will therefore not be picked up
• deviations can be in multiple folders at once, so you would be downloading the same image multiple times

As an example, take this account: https://www.deviantart.com/justatest235723 with two posted images.

• https://www.deviantart.com/justatest235723/art/Test-798674877 is in two folders
• https://www.deviantart.com/justatest235723/art/Unlisted-799178754 is in none

flat=true will give you both, but flat=false will only download the first one - twice.

I would be possible to add a folders metadata field that lists all folder names a deviation is part of, but that might cost a bit of network traffic and API calls if used.

[ReneRanzinger]

I would not mind the network traffic. Since the only other option to reconstruct the folder AND get all images is to download everything twice. Once with flat=true and once with flat=false. After this I can run a de-duplicate file tool and remove the "folder images" from the flat folder. If you are worried about the additional traffic why not make it an option in the configuration file that is false by default?

…

________________________________________ From: Mike Fährmann <[email protected]> Sent: Sunday, May 26, 2019 3:08:41 PM To: mikf/gallery-dl Cc: Rene Ranzinger; Author Subject: Re: [mikf/gallery-dl] Reconstructing DeviantArt gallery/folder structure (#276) There is currently no such option, and flat will be fixed when f837ea9<f837ea9> is released in the next version. flat=false works by going through the list of all gallery folders, but that has its downsides: * deviations can be in no folder at all and will therefore not be picked up * deviations can be in multiple folders at once, so you would be downloading the same image multiple times As an example, take this account: https://www.deviantart.com/justatest235723 with two posted images. * https://www.deviantart.com/justatest235723/art/Test-798674877 is in two folders * https://www.deviantart.com/justatest235723/art/Unlisted-799178754 is in none flat=true will give you both, but flat=false will only download the first one - twice. I would be possible to add a folders metadata field that lists all folder names a deviation is part of, but that might cost a bit of network traffic and API calls if used. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#276?email_source=notifications&email_token=AARJT6DIM4TN5YUG4I3VZBTPXLN3TA5CNFSM4HOVLTHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWILSCA#issuecomment-496023816>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AARJT6ESWOVXYW64YRC2LSDPXLN3TANCNFSM4HOVLTHA>.

[Hrxn]

Since the only other option to reconstruct the folder AND get all images is to download everything twice.

Depends, I think. Using the archive option together with skip = true should work as well.

[mikf]

That's a really nice idea I didn't even think about, but it sadly doesn't work. Initially the archive IDs for flat=true and flat=false are different (one uses the gallery extractor, the other the folder extractor), and when you adjust them to match you'd be skipping files that are in more than 1 folder ...

Anyway @ReneRanzinger , I added a folders option that will add a list of folder names to each deviation. You can download all files with flat=true, write their metadata to separate files, and use those folder names to put them where they belong afterwards.

Parent folders are indicated with a /:

 "folders": [
    "Featured",
    "Artworks/Variants"
],

(Technically each folder is a child of Featured, but that is ignored)

[YukiUnHappy]

Can the deviantart extractor process the specified folder and recursive process the sub folder then download in the sub directory? Just like to keep the multilevel folder structures as the author set.

[Hrxn]

Maybe a new option for the postprocessor? 😄

[ReneRanzinger]

@mikf Thanks for implementing the option. I tested it and works great.