[twitter] Images from replies are broken

[ponchojohn1234]

Images from replies no longer appear on the /media tab, therefore they don't get downloaded, it can be worked around by using /with_replies but that downloads way more than necessary even with retweets=false, also makes it hit the rate limit faster.

[mikf]

You can manually set strategy to use the regular "tweets" timeline when you need replies.

[ponchojohn1234]

it doesn't really work, it still misses media from tweets that are replies, and the problem with /with_replies is that it downloads tweets that aren't from the user that i want to download, is there a way to avoid downloading files that don't match the URL user?

[mikf]

Set replies to "self"
or use image-filter with something like "author is user" as condition.

[Vladimir-russian]

Hi @mikf
I want to download everything ( media, RTs Fav etc) from a profile. Is this setting good ?

"twitter":
{
"username": "abc",
"password": "123",
"cards": false,
"conversations": false,
"pinned": true,
"quoted": false,
"replies": true,
"retweets": true,
"strategy": "with_replies, media",
"text-tweets": false,
"twitpic": false,
"unique": true,

[a84r7a3rga76fg]

@Vladimir-russian
Add this too

"include": ["all"],

[Vladimir-russian]

@Vladimir-russian Add this too

"include": ["all"],

do I need [] this ? Because GAL-D doesn't work with this but if I removed [], it works

"include": "all"

[a84r7a3rga76fg]

Yeah, seems [] wasn't necessary.

[ponchojohn1234]

Set replies to "self" or use image-filter with something like "author is user" as condition.

image-filter works, thanks

[CatPlanet]

Hi, I use following config to download images/videos and I still have to manually use /with_replies endpoint instead of old /media because media one skips a lot of tweets

"twitter":
        {
	    "skip": "abort:50",
            "username": "...",
            "password": "...",
            "cards": false,
            "conversations": false,
            "quoted": false,
            "replies": true,
            "retweets": false,
            "text-tweets": false,
            "twitpic": false,
            "users": "timeline",
            "videos": true
        }

am I doing something wrong?

[mikf]

@CatPlanet Set strategy to "with_replies".

[CatPlanet]

@mikf I've skipped not relevant and/or defaults and I ended with

"strategy": "with_replies",
"replies": true,
"image-filter": "author is user",

just to emulate previous behavior. If only I knew this tiny "improvements" note in changelog would have such impact on application I would stop using it for a while instead of spending next few weeks getting rate limited 😣 I just wonder how many users as me will notice they're getting partial results.

Anyway have a nice day, I love your project.

[AlttiRi]

It seems

"replies": "self"

is ignored when I pass -o text-tweets=true.

I now get text tweets with replies to other users.

[MarqFJA87]

@mikf I've skipped not relevant and/or defaults and I ended with

"strategy": "with_replies",
"replies": true,
"image-filter": "author is user",

just to emulate previous behavior. If only I knew this tiny "improvements" note in changelog would have such impact on application I would stop using it for a while instead of spending next few weeks getting rate limited 😣 I just wonder how many users as me will notice they're getting partial results.

Anyway have a nice day, I love your project.

You sure this setup is correct? It gave me a much smaller portion of the account's media tweets than I had expected (just in the double digits). I had to add ", media" to the strategy parameter to get a little over 500 media tweets before the process concluded, and even then I'm not sure if that's because the account is under a shadowban or if it's because there's still room for expanding the scope of the search.

[CatPlanet]

You sure this setup is correct?

Only if defaults from this page are correct, yes. "replies" is also defaulting to true but I felt like adding it to my post too for clarity's sake. My full config is as follows:

"twitter":
{
	"skip": "true",
	"ratelimit": "wait",
        "strategy": "with_replies",
	"username": "aaa",
	"password": "bbb",
	"cards": false,
	"conversations": false,
	"quoted": false,
	"replies": true,
	"image-filter": "author is user",
	"retweets": false,
	"text-tweets": false,
	"twitpic": false,
	"users": "timeline",
	"videos": true
},

I'm 90% sure this is correct setup; I haven't noticed anything missing (yet) and I get rate-limited pretty fast so 🙂
(skip:true is temporary, for fixing what's been missed)

[mikf]

Only if defaults from this page are correct

replies are in fact enabled by default:

gallery-dl/gallery_dl/extractor/twitter.py

Line 40 in f954419

self.replies = self.config("replies", True)

It seems

"replies": "self"

is ignored when I pass -o text-tweets=true.

I now get text tweets with replies to other users.

"replies": "self" should work the same for text Tweets as for media Tweets.
Have you tried "image-filter": "author is user"?

[AlttiRi]

"replies": "self" is ignored when I pass -o text-tweets=true.

Have you tried "image-filter": "author is user"?

No, I just though the program could work without it, as before.

[CatPlanet]

At this point I think everyone would appreciate definitive config setup to get every media piece from given user.
@mikf would you kindly provide one to us?

[mikf]

I would if I could, but I also do not know how to get "all" Tweets or if that's even possible. If I knew of a way I would most certainly make this the default setting.

[MarqFJA87]

My way - which I described here seems to have more or less the same effect as the old status quo before the recent changes on Twitter's end borked it.

[mikf]

If I understand your comment right and you mean setting strategy to "with_replies, media", then that sadly doesn't work.

Using an invalid/unrecognized value for strategy, which "with_replies, media" is, makes gallery-dl silently fall back to a default value - /media in this case - and Tweets from that are incomplete since they don't include replies anymore.

[MarqFJA87]

Dammit, you're right, I just tested it with one of the accounts whose media timeline was severely demolished by the "no replies in the media timeline" BS, and it really doesn't work.

EDIT 1: Currently trying the "include" parameter with the setting "tweets,media,replies". So far it seems to have the desired effect; it's not stopping at the double digits like when I first tried the suggested method here.

EDIT 2: Well, here's an interesting observation. It goes through each of those categories one by one, instead of my expectation that it they're criteria to check tweets against. In hindsight, my initial assumption was obviously stupid.

EDIT 3: Alright, it's done. End result is that it grabbed a lot, but unfortunately is missing much of the older stuff despite it still being provably present. I cannot tell if this is a problem unique to this account or if it's a general issue. One weird anomaly, however, is that it grabbed the text captions (but not the images) of several tweets by other accounts; the only tying factor that I could see is that they were retweeted by the actual target account.

[Fukitsu]

Is it me or are replies back in the media tab?

[mikf]

Same here. Media replies appear to get listed in /media again,
although out of order nvm, that was because of unique and other options.

[MarqFJA87]

They seem to be in the correct order for me so far. Maybe this only affects some accounts, whether it's on the observer's end or the observed party's end (well, technically on Twitter's end when it fetches the data about that account).

Can anyone confirm if this has affected gallery-dl's ability to fetch media tweets?

[Fukitsu]

It hasn't but what I noticed is that it will grab media replies regardless of "replies": "self" or "image-filter": "author is user",

[mikf]

Not sure I understand the problem here. /media contains only Tweets from a particular user, at least I think/hope it does, so "replies": "self" and "image-filter": "author is user", should not be filtering out anything.

[MarqFJA87]

I just tested it, and can verify that currently the app misses a very significant portion of a Twitter account's media tweets (over 80 tweets to be specific, out of a total 188 as of this writing). And this is despite the fact that I haven't changed my settings from before the media timeline was "fixed" three days ago.

[mikf]

@MarqFJA87
I'd need a more detailed explanation as to what exactly you are doing, e.g. input URL, your Twitter settings, --verbose log, etc to figure out what's wrong and fix it.
It might just be related to it using /tweets instead of /media by default in v1.26.5 and .6 (a94f944).

[MarqFJA87]

@mikf Sure.

Input URL: https://twitter.com/strangestone (quite a few mildly to very NSFW images, given the artist's main work is unashamedly fanservice-based)

Twitter settings (omitting username, password and cookie entries), which I have been using to deal with the preceding status quo:

            "cards": false,
            "conversations": false,
            "image-filter": "author is user",
            "logout": true,
            "pinned": false,
            "quoted": false,
            "replies": true,
            "retweets": false,
            "text-tweets": false,
            "twitpic": false,
            "users": "timeline",
            "videos": "ytdl",

The verbose log is over 2000 lines long, so I'll be attaching it as a text file.
gallery-dl verbose log.txt

Just to illustrate, the app is missing tweets as recent as this one.

[mikf]

@MarqFJA87 Like I thought, it is indeed using the /tweets timeline (.../UserTweets?variables=...) and not /media.
I'm not completely done with testing everything on my own machine, but adding "strategy": "media" to your config will help and at the very least download tweet 1713822791542391146.
(commit a94f944 was probably a mistake. it'll be reverted in the next release 5c43098)

[MarqFJA87]

Thanks. I will report back if I notice that this method is still somehow missing some tweets.

[mikf]

I ran your URL/config with both "strategy": "tweets" (what you've implicitly been using) and "strategy": "media", and the difference is "only" 34 files more.

File Diff

1744132052486619385_1.png
1728979297468317925_2.jpg
1728979297468317925_1.jpg
1713822791542391146_1.png
1680814663175659521_1.png
1673212921114144774_1.png
1510491123185201153_1.jpg
1432156524902715392_1.png
1425405459209494535_1.jpg
1379381103836942336_1.png
1361273519573594113_1.png
1310414064380641280_1.png
1257234862689316866_1.jpg
1252068110972026882_1.png
1193676606893518848_1.png
1193665231269986304_1.png
1183518023862247424_1.png
1180998161524908032_1.png
1101859093138948099_1.png
1079512706921639936_1.png
1069388060905791488_1.png
929860005821431808_1.png
917165629613977600_1.png
895804122603020288_1.jpg
892695460153933828_1.png
884197374716821508_1.png
841063459399729152_1.png
818237049136234496_1.png
764098971132317696_1.png
706914185553137664_1.jpg
652075868228067328_1.png
619841635703128064_1.jpg
604626418618658816_1.png
592731606659506177_1.png