Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Weasyl] Submission extractor metadata is incomplete when downloading whole gallery #2610

Closed
tux93 opened this issue May 20, 2022 · 2 comments
Closed

[Weasyl] Submission extractor metadata is incomplete when downloading whole gallery #2610

tux93 opened this issue May 20, 2022 · 2 comments
Labels
enhancement site:feature

Comments

@tux93
Copy link
Contributor

tux93 commented May 20, 2022 •
edited
Loading

I noticed that the metadata for submissions is incomplete when downloading a users whole gallery, for example the Description field is missing:

~> gallery-dl -vK https://www.weasyl.com/~kiaun
[gallery-dl][debug] Version 1.21.2
[gallery-dl][debug] Python 3.8.13 - Linux-5.17.7-1-default-x86_64-with-glibc2.34
[gallery-dl][debug] requests 2.26.0 - urllib3 1.26.6
[gallery-dl][debug] Starting KeywordJob for 'https://www.weasyl.com/~kiaun'
[weasyl][debug] Using WeasylSubmissionsExtractor for 'https://www.weasyl.com/~kiaun'
Keywords for directory names:
-----------------------------
category
  weasyl
owner_login
  kiaun
subcategory
  submissions
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.weasyl.com:443
[urllib3.connectionpool][debug] https://www.weasyl.com:443 "GET /api/users/kiaun/gallery HTTP/1.1" 200 None

Keywords for filenames and --filter:
------------------------------------
category
  weasyl
date
  2022-05-17 16:51:59
extension
  jpg
filename
  kiaun-aden-and-alpha
folderid
  None
link
  https://www.weasyl.com/submission/2128334/aden-and-alpha
media[cover][][mediaid]
  8056541
media[cover][][url]
  https://cdn.weasyl.com/static/media/35/5c/3e/355c3e1908b20f412d7b07a234e89e5acf53547484f44fd6ef100f7b5d9f9948.jpg
media[submission][][mediaid]
  8056540
media[submission][][url]
  https://cdn.weasyl.com/~kiaun/submissions/2128334/2b9aa39da60655a1dd642728c1acf121ac667756db272b32df7a549568c369f0/kiaun-aden-and-alpha.jpg
media[thumbnail][][mediaid]
  8056542
media[thumbnail][][url]
  https://cdn.weasyl.com/static/media/11/a4/81/11a4816bb55b9698ab48e905ad0b5414f05bb577c8bd1f5c89f853bd5d68f393.jpg
media[thumbnail-generated][][mediaid]
  8056542
media[thumbnail-generated][][url]
  https://cdn.weasyl.com/static/media/11/a4/81/11a4816bb55b9698ab48e905ad0b5414f05bb577c8bd1f5c89f853bd5d68f393.jpg
media[thumbnail-generated-webp][][mediaid]
  8056543
media[thumbnail-generated-webp][][url]
  https://cdn.weasyl.com/static/media/e9/c6/03/e9c603d087e740f905fc152492b6f18d74e2196f3a1e2fe0d88f22413a10bea3.webp
owner
  Kiaun
owner_login
  kiaun
posted_at
  2022-05-17T16:51:59Z
rating
  general
subcategory
  submissions
submitid
  2128334
subtype
  visual
title
  Aden & Alpha
type
  submission
url
  https://cdn.weasyl.com/~kiaun/submissions/2128334/2b9aa39da60655a1dd642728c1acf121ac667756db272b32df7a549568c369f0/kiaun-aden-and-alpha.jpg

When using the direct link to a submission the Description field is present:

~> gallery-dl -vK https://www.weasyl.com/~kiaun/submissions/2128334/aden-and-alpha
[gallery-dl][debug] Version 1.21.2
[gallery-dl][debug] Python 3.8.13 - Linux-5.17.7-1-default-x86_64-with-glibc2.34
[gallery-dl][debug] requests 2.26.0 - urllib3 1.26.6
[gallery-dl][debug] Starting KeywordJob for 'https://www.weasyl.com/~kiaun/submissions/2128334/aden-and-alpha'
[weasyl][debug] Using WeasylSubmissionExtractor for 'https://www.weasyl.com/~kiaun/submissions/2128334/aden-and-alpha'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.weasyl.com:443
[urllib3.connectionpool][debug] https://www.weasyl.com:443 "GET /api/submissions/2128334/view HTTP/1.1" 200 None
Keywords for directory names:
-----------------------------
category
  weasyl
comments
  1
date
  2022-05-17 16:51:59
description
  <div><p>since I made a drawing of Ivory and Laura, I felt this one was also worth making! heck yes!<br>I can see people who like Aden more than Alpha, and other that probably doesn't know exactly who's Alpha, since they've been reading the remake and didn't read the original comic, but don't worry! we'll get to new episodes and updates quickly with a the two updates per week format.</p></div><div><br></div>
<div><p>read this comic at <a href="https://derideal.com/derideal" rel="nofollow ugc">https://derideal.com/derideal</a>  !  updates now Monday and Wednesday!</p></div><div><br></div>
<div><p><strong>Support Derideal in Patreon!</strong><br><a href="https://www.patreon.com/derideal" rel="nofollow ugc">https://www.patreon.com/derideal</a><br><br>
<br><br>
<strong>Tool used:</strong> ipad + procreate<br><br><br>
</p><div><strong>Useful links to find me! </strong><br><a href="https://www.furaffinity.net/user/kiaun/" rel="nofollow ugc">https://www.furaffinity.net/user/kiaun/</a><br><a href="https://www.deviantart.com/kiaun" rel="nofollow ugc">https://www.deviantart.com/kiaun</a><br><a href="https://twitter.com/derideal" rel="nofollow ugc">https://twitter.com/derideal</a><br><a href="https://www.instagram.com/deridealofficial/" rel="nofollow ugc">https://www.instagram.com/deridealofficial/</a></div><div><br></div>

<p></p><div>-<br><strong>READ ALL DERIDEAL COMICS:</strong><br><a href="https://derideal.com/comics" rel="nofollow ugc">https://derideal.com/comics</a></div><div><br></div><br><br><br><a href="http://www.postybirb.com/" rel="nofollow ugc">Posted using PostyBirb</a></div><br><br><p><a href="http://www.postybirb.com" rel="nofollow ugc">Posted using PostyBirb</a></p>

embedlink
  None
extension
  jpg
favorited
  False
favorites
  6
filename
  kiaun-aden-and-alpha
folder_name
  Derideal themed
folderid
  20834
friends_only
  False
link
  https://www.weasyl.com/submission/2128334/aden-and-alpha
media[cover][][mediaid]
  8056541
media[cover][][url]
  https://cdn.weasyl.com/static/media/35/5c/3e/355c3e1908b20f412d7b07a234e89e5acf53547484f44fd6ef100f7b5d9f9948.jpg
media[submission][][mediaid]
  8056540
media[submission][][url]
  https://cdn.weasyl.com/~kiaun/submissions/2128334/2b9aa39da60655a1dd642728c1acf121ac667756db272b32df7a549568c369f0/kiaun-aden-and-alpha.jpg
media[thumbnail][][mediaid]
  8056542
media[thumbnail][][url]
  https://cdn.weasyl.com/static/media/11/a4/81/11a4816bb55b9698ab48e905ad0b5414f05bb577c8bd1f5c89f853bd5d68f393.jpg
media[thumbnail-generated][][mediaid]
  8056542
media[thumbnail-generated][][url]
  https://cdn.weasyl.com/static/media/11/a4/81/11a4816bb55b9698ab48e905ad0b5414f05bb577c8bd1f5c89f853bd5d68f393.jpg
media[thumbnail-generated-webp][][mediaid]
  8056543
media[thumbnail-generated-webp][][url]
  https://cdn.weasyl.com/static/media/e9/c6/03/e9c603d087e740f905fc152492b6f18d74e2196f3a1e2fe0d88f22413a10bea3.webp
owner
  Kiaun
owner_login
  kiaun
owner_media[avatar][][mediaid]
  951346
owner_media[avatar][][url]
  https://cdn.weasyl.com/static/media/8c/62/16/8c62163cb023c50c9bf1cdb125cce5262114205df77f41798f47af687b8176a7.png
owner_media[banner][][mediaid]
  951345
owner_media[banner][][url]
  https://cdn.weasyl.com/static/media/80/63/7d/80637dad1e99e6dfc1f1eb01cf70eafd8fbb51b656892a18708fc3e294c21851.png
posted_at
  2022-05-17T16:51:59+00:00Z
rating
  general
subcategory
  submission
submitid
  2128334
subtype
  visual
tags[]
  - anthro
  - cat
  - cats
  - comic
  - comics
  - consortium
  - derideal
  - furry
  - oc
  - sketch
  - webcomics
  - xeno
  - xenos
title
  Aden & Alpha
type
  submission
url
  https://cdn.weasyl.com/~kiaun/submissions/2128334/2b9aa39da60655a1dd642728c1acf121ac667756db272b32df7a549568c369f0/kiaun-aden-and-alpha.jpg
views
  29

Keywords for filenames and --filter:
------------------------------------
category
  weasyl
comments
  1
date
  2022-05-17 16:51:59
description
  <div><p>since I made a drawing of Ivory and Laura, I felt this one was also worth making! heck yes!<br>I can see people who like Aden more than Alpha, and other that probably doesn't know exactly who's Alpha, since they've been reading the remake and didn't read the original comic, but don't worry! we'll get to new episodes and updates quickly with a the two updates per week format.</p></div><div><br></div>
<div><p>read this comic at <a href="https://derideal.com/derideal" rel="nofollow ugc">https://derideal.com/derideal</a>  !  updates now Monday and Wednesday!</p></div><div><br></div>
<div><p><strong>Support Derideal in Patreon!</strong><br><a href="https://www.patreon.com/derideal" rel="nofollow ugc">https://www.patreon.com/derideal</a><br><br>
<br><br>
<strong>Tool used:</strong> ipad + procreate<br><br><br>
</p><div><strong>Useful links to find me! </strong><br><a href="https://www.furaffinity.net/user/kiaun/" rel="nofollow ugc">https://www.furaffinity.net/user/kiaun/</a><br><a href="https://www.deviantart.com/kiaun" rel="nofollow ugc">https://www.deviantart.com/kiaun</a><br><a href="https://twitter.com/derideal" rel="nofollow ugc">https://twitter.com/derideal</a><br><a href="https://www.instagram.com/deridealofficial/" rel="nofollow ugc">https://www.instagram.com/deridealofficial/</a></div><div><br></div>

<p></p><div>-<br><strong>READ ALL DERIDEAL COMICS:</strong><br><a href="https://derideal.com/comics" rel="nofollow ugc">https://derideal.com/comics</a></div><div><br></div><br><br><br><a href="http://www.postybirb.com/" rel="nofollow ugc">Posted using PostyBirb</a></div><br><br><p><a href="http://www.postybirb.com" rel="nofollow ugc">Posted using PostyBirb</a></p>

embedlink
  None
extension
  jpg
favorited
  False
favorites
  6
filename
  kiaun-aden-and-alpha
folder_name
  Derideal themed
folderid
  20834
friends_only
  False
link
  https://www.weasyl.com/submission/2128334/aden-and-alpha
media[cover][][mediaid]
  8056541
media[cover][][url]
  https://cdn.weasyl.com/static/media/35/5c/3e/355c3e1908b20f412d7b07a234e89e5acf53547484f44fd6ef100f7b5d9f9948.jpg
media[submission][][mediaid]
  8056540
media[submission][][url]
  https://cdn.weasyl.com/~kiaun/submissions/2128334/2b9aa39da60655a1dd642728c1acf121ac667756db272b32df7a549568c369f0/kiaun-aden-and-alpha.jpg
media[thumbnail][][mediaid]
  8056542
media[thumbnail][][url]
  https://cdn.weasyl.com/static/media/11/a4/81/11a4816bb55b9698ab48e905ad0b5414f05bb577c8bd1f5c89f853bd5d68f393.jpg
media[thumbnail-generated][][mediaid]
  8056542
media[thumbnail-generated][][url]
  https://cdn.weasyl.com/static/media/11/a4/81/11a4816bb55b9698ab48e905ad0b5414f05bb577c8bd1f5c89f853bd5d68f393.jpg
media[thumbnail-generated-webp][][mediaid]
  8056543
media[thumbnail-generated-webp][][url]
  https://cdn.weasyl.com/static/media/e9/c6/03/e9c603d087e740f905fc152492b6f18d74e2196f3a1e2fe0d88f22413a10bea3.webp
owner
  Kiaun
owner_login
  kiaun
owner_media[avatar][][mediaid]
  951346
owner_media[avatar][][url]
  https://cdn.weasyl.com/static/media/8c/62/16/8c62163cb023c50c9bf1cdb125cce5262114205df77f41798f47af687b8176a7.png
owner_media[banner][][mediaid]
  951345
owner_media[banner][][url]
  https://cdn.weasyl.com/static/media/80/63/7d/80637dad1e99e6dfc1f1eb01cf70eafd8fbb51b656892a18708fc3e294c21851.png
posted_at
  2022-05-17T16:51:59+00:00Z
rating
  general
subcategory
  submission
submitid
  2128334
subtype
  visual
tags[]
  - anthro
  - cat
  - cats
  - comic
  - comics
  - consortium
  - derideal
  - furry
  - oc
  - sketch
  - webcomics
  - xeno
  - xenos
title
  Aden & Alpha
type
  submission
url
  https://cdn.weasyl.com/~kiaun/submissions/2128334/2b9aa39da60655a1dd642728c1acf121ac667756db272b32df7a549568c369f0/kiaun-aden-and-alpha.jpg
views
  29

Since I'm using a metadata post processor to save descriptions this breaks gallery downloads:

~> gallery-dl -v https://www.weasyl.com/~kiaun
[gallery-dl][debug] Version 1.21.2
[gallery-dl][debug] Python 3.8.13 - Linux-5.17.7-1-default-x86_64-with-glibc2.34
[gallery-dl][debug] requests 2.26.0 - urllib3 1.26.6
[gallery-dl][debug] Starting DownloadJob for 'https://www.weasyl.com/~kiaun'
[weasyl][debug] Using WeasylSubmissionsExtractor for 'https://www.weasyl.com/~kiaun'
[weasyl][debug] Using download archive '[...]/archive-weasyl.sqlite3'
[weasyl][debug] Active postprocessor modules: [MetadataPP, MtimePP]
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.weasyl.com:443
[urllib3.connectionpool][debug] https://www.weasyl.com:443 "GET /api/users/kiaun/gallery HTTP/1.1" 200 None
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): cdn.weasyl.com:443
[urllib3.connectionpool][debug] https://cdn.weasyl.com:443 "GET /~kiaun/submissions/2128334/2b9aa39da60655a1dd642728c1acf121ac667756db272b32df7a549568c369f0/kiaun-aden-and-alpha.jpg HTTP/1.1" 200 361602
[...]/Weasyl/kiaun/2128334_kiaun-aden-and-alpha.jpg
[weasyl][error] An unexpected error occurred: NameError - name 'description' is not defined. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[weasyl][debug] 
Traceback (most recent call last):
  File "/home/fl4nn/.venv/gallery-dl/lib64/python3.8/site-packages/gallery_dl/job.py", line 77, in run
    self.dispatch(msg)
  File "/home/fl4nn/.venv/gallery-dl/lib64/python3.8/site-packages/gallery_dl/job.py", line 121, in dispatch
    self.handle_url(url, kwdict)
  File "/home/fl4nn/.venv/gallery-dl/lib64/python3.8/site-packages/gallery_dl/job.py", line 273, in handle_url
    callback(pathfmt)
  File "/home/fl4nn/.venv/gallery-dl/lib64/python3.8/site-packages/gallery_dl/job.py", line 496, in _call_hook
    if condition(pathfmt.kwdict):
  File "<expr>", line 1, in <module>
NameError: name 'description' is not defined
mikf added a commit that referenced this issue May 20, 2022
@mikf
[weasyl] implement 'metadata' option (#2610)
79dce8a
@mikf
Copy link
Owner

mikf commented May 20, 2022

There is now a metadata option for weasyl, that, when enabled, does one extra API request for each submission to fetch its full metadata. (79dce8a)

@tux93
Copy link
Contributor Author

tux93 commented May 25, 2022

Fixed with 1.22.0

@tux93 tux93 closed this as completed May 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement site:feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tux93 @mikf