Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[twitter] Syndication errors with key error #2499

Closed
shinji257 opened this issue Apr 15, 2022 · 0 comments
Closed

[twitter] Syndication errors with key error #2499

shinji257 opened this issue Apr 15, 2022 · 0 comments
Labels

Comments

@shinji257
Copy link
Contributor

shinji257 commented Apr 15, 2022

Example account: https://nitter.net/TheStarryOuija
Be aware the profile feed is very much NSFW.

Twitter config excerpt:

        "twitter":
        {
            "quoted": false,
            "conversations": false,
            "cards": true,
            "replies": "self",
            "retweets": "original",
            "twitpic": true,
            "videos": true,
            "logout": true,
            "skip": true,
            "syndication": true,
            "cookies": {
                "auth_token": "(redacted)"
            }

From what I can see on an alternative account this is a retweet that it is erroring on.

Here is the output. I skipped a bunch of file downloads since it seemed to be working...

>  .\gallery-dl.exe https://www.nitter.net/TheStarryOuija --verbose
[gallery-dl][debug] Version 1.21.1 - Executable
[gallery-dl][debug] Python 3.7.9 - Windows-10-10.0.22593
[gallery-dl][debug] requests 2.27.1 - urllib3 1.26.9
[gallery-dl][debug] Starting DownloadJob for 'https://www.nitter.net/TheStarryOuija'
[twitter][debug] Using TwitterTimelineExtractor for 'https://www.nitter.net/TheStarryOuija'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/7mjxD3-C6BxitPMVQ6w0-Q/UserByScreenName?variables=%7B%22screen_name%22%3A%22TheStarryOuija%22%2C%22withSafetyModeUserFields%22%3Atrue%2C%22withSuperFollowsUserFields%22%3Atrue%7D HTTP/1.1" 200 1166
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/WZT7sCTrLvSOaWOXLDsWbQ/UserTweets?variables=%7B%22userId%22%3A%221029413593563914240%22%2C%22count%22%3A100%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22includePromotedContent%22%3Afalse%2C%22withSuperFollowsUserFields%22%3Atrue%2C%22withBirdwatchPivots%22%3Afalse%2C%22withDownvotePerspective%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%2C%22withSuperFollowsTweetFields%22%3Atrue%2C%22withClientEventToken%22%3Afalse%2C%22withBirdwatchNotes%22%3Afalse%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Afalse%2C%22__fs_interactive_text%22%3Afalse%2C%22__fs_dont_mention_me_view_api_enabled%22%3Afalse%7D HTTP/1.1" 200 80
[twitter][debug] {'user': {'result': {'__typename': 'UserUnavailable'}}}
[twitter][info] Retrying API request as guest
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/WZT7sCTrLvSOaWOXLDsWbQ/UserTweets?variables=%7B%22userId%22%3A%221029413593563914240%22%2C%22count%22%3A100%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22includePromotedContent%22%3Afalse%2C%22withSuperFollowsUserFields%22%3Atrue%2C%22withBirdwatchPivots%22%3Afalse%2C%22withDownvotePerspective%22%3Afalse%2C%22withReactionsMetadata%22%3Afalse%2C%22withReactionsPerspective%22%3Afalse%2C%22withSuperFollowsTweetFields%22%3Atrue%2C%22withClientEventToken%22%3Afalse%2C%22withBirdwatchNotes%22%3Afalse%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Afalse%2C%22__fs_interactive_text%22%3Afalse%2C%22__fs_dont_mention_me_view_api_enabled%22%3Afalse%7D HTTP/1.1" 200 61712
[twitter][debug] Using download archive './gallery-dl/.archives/twitter.sqlite3'
(skipping lines...)
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): cdn.syndication.twimg.com:443
[urllib3.connectionpool][debug] https://cdn.syndication.twimg.com:443 "GET /tweet?id=1510676189811077125 HTTP/1.1" 200 662
[twitter][error] An unexpected error occurred: KeyError - 'user_id_str'. Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[twitter][debug]
Traceback (most recent call last):
  File "gallery_dl\job.pyc", line 76, in run
  File "gallery_dl\extractor\twitter.pyc", line 78, in items
KeyError: 'user_id_str'

Here is the JSON output I got when I went to the syndication url directly.

{"in_reply_to_screen_name":"HurleysArt","in_reply_to_user_id_str":"823727605299970049","lang":"und","favorite_count":652,"created_at":"2022-04-01T11:05:07.000Z","display_text_range":[0,16],"entities":{"hashtags":[],"urls":[],"user_mentions":[{"id_str":"823727605299970049","indices":[0,11],"name":"HurleysArt 🔞","screen_name":"HurleysArt"}],"symbols":[],"media":[{"display_url":"pic.twitter.com/v20Hig4ztF","expanded_url":"https://twitter.com/Creature_Candy/status/1509849335911141377/photo/1","indices":[15,38],"url":"https://t.co/v20Hig4ztF"}]},"id_str":"1509849335911141377","text":"@HurleysArt 😈💕 https://t.co/v20Hig4ztF","user":{"id_str":"4895553585","name":"Ashe 🖤☠","profile_image_url_https":"https://pbs.twimg.com/profile_images/1394628833354166278/3Magn_XE_normal.jpg","screen_name":"Creature_Candy","verified":false},"photos":[],"video":{"aspectRatio":[500,603],"contentType":"gif","durationMs":0,"mediaAvailability":{"status":"available"},"poster":"https://pbs.twimg.com/tweet_video_thumb/FPQP8O-XwAMvqiI.jpg","variants":[{"type":"video/mp4","src":"https://video.twimg.com/tweet_video/FPQP8O-XwAMvqiI.mp4"}],"videoId":{"type":"tweet","id":"1509849335911141377"},"viewCount":0},"conversation_count":6,"news_action_type":"conversation"}
@shinji257 shinji257 changed the title Syndication errors with key error [twitter] Syndication errors with key error Apr 15, 2022
mikf added a commit that referenced this issue Apr 21, 2022
- handle retweets
- fix videos without dimensions in URL (3e942a5)
- fix '"retweets": "self"' filter (#2499)
@mikf mikf added the bug label Dec 4, 2022
@mikf mikf closed this as completed Dec 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants