[twitter] Guest tokens seemingly no longer work with python-requests 2.32.3

[kattjevfel]

Trying to get publicly viewable posts used to work without an account with gallery-dl, but seemingly no more since today.

katt@main ~ » gallery-dl --config-ignore --verbose https://x.com/als9786/status/1791139457812545605 
[gallery-dl][debug] Version 1.27.0-dev
[gallery-dl][debug] Python 3.12.3 - Linux-6.9.2-zen1-1-zen-x86_64-with-glibc2.39
[gallery-dl][debug] requests 2.32.3 - urllib3 1.26.18
[gallery-dl][debug] Configuration Files []
[gallery-dl][debug] Starting DownloadJob for 'https://x.com/als9786/status/1791139457812545605'
[twitter][debug] Using TwitterTweetExtractor for 'https://x.com/als9786/status/1791139457812545605'
[twitter][info] Requesting guest token
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): api.x.com:443
[twitter][debug] HTTPSConnectionPool(host='api.x.com', port=443): Max retries exceeded with url: /1.1/guest/activate.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))) (1/5)
[twitter][debug] Sleeping 1.00 seconds (retry)
[urllib3.connectionpool][debug] Starting new HTTPS connection (2): api.x.com:443
[twitter][debug] HTTPSConnectionPool(host='api.x.com', port=443): Max retries exceeded with url: /1.1/guest/activate.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))) (2/5)
[twitter][debug] Sleeping 2.00 seconds (retry)
[urllib3.connectionpool][debug] Starting new HTTPS connection (3): api.x.com:443
[twitter][debug] HTTPSConnectionPool(host='api.x.com', port=443): Max retries exceeded with url: /1.1/guest/activate.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))) (3/5)
[twitter][debug] Sleeping 3.00 seconds (retry)
[urllib3.connectionpool][debug] Starting new HTTPS connection (4): api.x.com:443
[twitter][debug] HTTPSConnectionPool(host='api.x.com', port=443): Max retries exceeded with url: /1.1/guest/activate.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))) (4/5)
[twitter][debug] Sleeping 4.00 seconds (retry)
[urllib3.connectionpool][debug] Starting new HTTPS connection (5): api.x.com:443
[twitter][debug] HTTPSConnectionPool(host='api.x.com', port=443): Max retries exceeded with url: /1.1/guest/activate.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))) (5/5)
[twitter][error] HttpError: SSLError: HTTPSConnectionPool(host='api.x.com', port=443): Max retries exceeded with url: /1.1/guest/activate.json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)')))

gallery-dl e353171

[mikf]

Your error seems to be more SSL certificate related than anything:

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate

Still works for me by the way, but I haven't really updated anything on my system in quite a while.

$ gallery-dl -v  https://x.com/als9786/status/1791139457812545605 
[gallery-dl][debug] Version 1.27.0-dev - Git HEAD: baee5977
[gallery-dl][debug] Python 3.11.8 - Linux-6.8.7-arch1-1-x86_64-with-glibc2.39
[gallery-dl][debug] requests 2.31.0 - urllib3 2.1.0
[gallery-dl][debug] Configuration Files ['${HOME}/.gallery-dl.conf']
[gallery-dl][debug] Starting DownloadJob for 'https://x.com/als9786/status/1791139457812545605'
[twitter][debug] Using TwitterTweetExtractor for 'https://x.com/als9786/status/1791139457812545605'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): x.com:443
[urllib3.connectionpool][debug] https://x.com:443 "GET /i/api/graphql/MWY3AO9_I3rcP_L2A4FR4A/TweetResultByRestId?variables=%7B%22tweetId%22%3A%221791139457812545605%22%2C%22withCommunity%22%3Afalse%2C%22includePromotedContent%22%3Afalse%2C%22withVoice%22%3Afalse%7D&features=%7B%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22tweetypie_unmention_optimization_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22rweb_video_timestamps_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_media_download_video_enabled%22%3Atrue%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D HTTP/1.1" 200 1447
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): pbs.twimg.com:443
[urllib3.connectionpool][debug] https://pbs.twimg.com:443 "GET /media/GNtmFqIbQAA9XkZ?format=jpg&name=orig HTTP/1.1" 200 185475
/tmp/_/twitter/als9786/1791139457812545605_1.jpg

[kattjevfel]

So I dug a tiny bit deeper and it seems like rolling back to python-requests 2.32.2 fixes it, changing back to 2.32.3 breaks it.

[mikf]

psf/requests#6730

Seems like psf/requests@b1d73dd broke or changed something,
i.e. it still works with the previous psf/requests@6badbac

[mikf]

Should be fixed with 6cfbc10

This new requests behavior was affecting all sites that used a custom SSLContext, like patreon or artstation, or whenever browser or tls12 options were used.

These options were entirely non-functional with requests 2.32.0, 2.32.1, and 2.32.2 by the way.

[kattjevfel]

Yup, can confirm it works with that commit. Thanks for the quick fix, even if it wasn't gallery-dl's fault, should probably keep an eye on that issue though in case it gets fixed upstream.

Thanks again!