Inconsistent results from Twitter; Apphending /media vs not generates vastly different results.

[pxssy]

My config has retweets and videos set to false, using the most recent version. With this setup i should only get images posted by the user.

Doing "gallery-dl https://twitter.com/amatou_0429" gets me 260 files.

Whereas "gallery-dl https://twitter.com/amatou_0429/media" gets me 332 files.

So it appears that the bare link without /media misses out some images. Not sure why. Now, i thought that everything in bare link would be included in /media, but i was wrong.

Doing a diff -r on the 2 folders

{ twitter } » diff -r './amatou_0429' './amatou_0429 media' | sort
Only in ./amatou_0429 media: 1010839366824910848_1.jpg
Only in ./amatou_0429 media: 1010839366824910848_2.jpg
Only in ./amatou_0429 media: 1010839366824910848_3.jpg
Only in ./amatou_0429 media: 1010839366824910848_4.jpg
Only in ./amatou_0429 media: 1039490811895799808_1.jpg
Only in ./amatou_0429 media: 1039490811895799808_2.jpg
Only in ./amatou_0429 media: 1039490811895799808_3.jpg
Only in ./amatou_0429 media: 1039490811895799808_4.jpg
Only in ./amatou_0429 media: 1040203863502946310_1.jpg
Only in ./amatou_0429 media: 1040203863502946310_2.jpg
Only in ./amatou_0429 media: 1040203863502946310_3.jpg
Only in ./amatou_0429 media: 1040203863502946310_4.jpg
Only in ./amatou_0429 media: 1040604144938057728_1.jpg
Only in ./amatou_0429 media: 1040919926494572544_1.jpg
Only in ./amatou_0429 media: 1040919926494572544_2.jpg
Only in ./amatou_0429 media: 1040919926494572544_3.jpg
Only in ./amatou_0429 media: 1040919926494572544_4.jpg
Only in ./amatou_0429 media: 1043828245328613376_1.jpg
Only in ./amatou_0429 media: 1043828245328613376_2.jpg
Only in ./amatou_0429 media: 1043828245328613376_3.jpg
Only in ./amatou_0429 media: 1043828245328613376_4.jpg
Only in ./amatou_0429 media: 1052524208490668032_1.jpg
Only in ./amatou_0429 media: 1060993626929475584_1.jpg
Only in ./amatou_0429 media: 1061202201907912705_1.jpg
Only in ./amatou_0429 media: 1061202201907912705_2.jpg
Only in ./amatou_0429 media: 1061202201907912705_3.jpg
Only in ./amatou_0429 media: 1063058356162650112_1.jpg
Only in ./amatou_0429 media: 1063058356162650112_2.jpg
Only in ./amatou_0429 media: 1063058356162650112_3.jpg
Only in ./amatou_0429 media: 1063058356162650112_4.jpg
Only in ./amatou_0429 media: 1065598488166129664_1.jpg
Only in ./amatou_0429 media: 1067384949316042753_1.jpg
Only in ./amatou_0429 media: 1067384949316042753_2.jpg
Only in ./amatou_0429 media: 1067384949316042753_3.jpg
Only in ./amatou_0429 media: 1067384949316042753_4.jpg
Only in ./amatou_0429 media: 1072471589181112320_1.jpg
Only in ./amatou_0429 media: 1072471589181112320_2.jpg
Only in ./amatou_0429 media: 1072471589181112320_3.jpg
Only in ./amatou_0429 media: 1072471589181112320_4.jpg
Only in ./amatou_0429 media: 1074292963591614467_1.jpg
Only in ./amatou_0429 media: 1074292963591614467_2.jpg
Only in ./amatou_0429 media: 1075163327922298880_1.jpg
Only in ./amatou_0429 media: 1075163327922298880_2.jpg
Only in ./amatou_0429 media: 1075163327922298880_3.jpg
Only in ./amatou_0429 media: 1075163327922298880_4.jpg
Only in ./amatou_0429 media: 1075362194181607424_1.jpg
Only in ./amatou_0429 media: 1075362194181607424_2.jpg
Only in ./amatou_0429 media: 1075362194181607424_3.jpg
Only in ./amatou_0429 media: 1075362194181607424_4.jpg
Only in ./amatou_0429 media: 1078239695367094272_1.jpg
Only in ./amatou_0429 media: 1079956156515110912_1.jpg
Only in ./amatou_0429 media: 1081513679697276928_1.jpg
Only in ./amatou_0429 media: 1081862985251844096_1.jpg
Only in ./amatou_0429 media: 1081862985251844096_2.jpg
Only in ./amatou_0429 media: 1081862985251844096_3.jpg
Only in ./amatou_0429 media: 1084220329600344065_1.jpg
Only in ./amatou_0429 media: 1089309410512363520_1.jpg
Only in ./amatou_0429 media: 1092003459358838785_1.jpg
Only in ./amatou_0429 media: 1092003459358838785_2.jpg
Only in ./amatou_0429 media: 1092003459358838785_3.jpg
Only in ./amatou_0429 media: 1095269140783190016_1.jpg
Only in ./amatou_0429 media: 1095269140783190016_2.jpg
Only in ./amatou_0429 media: 1097844035690016768_1.jpg
Only in ./amatou_0429 media: 1099109918139809792_1.jpg
Only in ./amatou_0429 media: 1099631863734231040_1.jpg
Only in ./amatou_0429 media: 1099631863734231040_2.jpg
Only in ./amatou_0429 media: 1099691353590575104_1.jpg
Only in ./amatou_0429 media: 1099691353590575104_2.jpg
Only in ./amatou_0429 media: 1099691353590575104_3.jpg
Only in ./amatou_0429 media: 1099691353590575104_4.jpg
Only in ./amatou_0429 media: 1099887892128620544_1.jpg
Only in ./amatou_0429 media: 1099887892128620544_2.jpg
Only in ./amatou_0429 media: 1099887892128620544_3.jpg
Only in ./amatou_0429 media: 1099887892128620544_4.jpg
Only in ./amatou_0429 media: 1102883107483443200_1.jpg
Only in ./amatou_0429 media: 1102883107483443200_2.jpg
Only in ./amatou_0429 media: 1102883107483443200_3.jpg
Only in ./amatou_0429 media: 1102883107483443200_4.jpg
Only in ./amatou_0429 media: 1103944727705968640_1.jpg
Only in ./amatou_0429 media: 1104285770783776768_1.jpg
Only in ./amatou_0429 media: 1106081264958824448_1.jpg
Only in ./amatou_0429 media: 1106136277500350464_1.jpg
Only in ./amatou_0429 media: 1106136277500350464_2.jpg
Only in ./amatou_0429 media: 1106136277500350464_3.jpg
Only in ./amatou_0429 media: 1106205926611025920_1.jpg
Only in ./amatou_0429 media: 1106205926611025920_2.jpg
Only in ./amatou_0429 media: 1107229575694606337_1.jpg
Only in ./amatou_0429 media: 1130052082025635840_1.jpg
Only in ./amatou_0429 media: 1130052082025635840_2.jpg
Only in ./amatou_0429 media: 1130052082025635840_3.jpg
Only in ./amatou_0429 media: 1130052082025635840_4.jpg
Only in ./amatou_0429 media: 901423677090045955_1.jpg
Only in ./amatou_0429 media: 901423677090045955_2.jpg
Only in ./amatou_0429 media: 901423677090045955_3.jpg
Only in ./amatou_0429 media: 901423677090045955_4.jpg
Only in ./amatou_0429 media: 922825343307628544_1.jpg
Only in ./amatou_0429 media: 946282831835099141_1.jpg
Only in ./amatou_0429 media: 954743084385841152_1.jpg
Only in ./amatou_0429 media: 955384545997570048_1.jpg
Only in ./amatou_0429 media: 955384545997570048_2.jpg
Only in ./amatou_0429 media: 955384545997570048_3.jpg
Only in ./amatou_0429 media: 955384545997570048_4.jpg
Only in ./amatou_0429 media: 978242736368246784_1.jpg
Only in ./amatou_0429: 1005791697526145024_1.jpg
Only in ./amatou_0429: 1027718609781514240_1.jpg
Only in ./amatou_0429: 1031481906628452352_1.jpg
Only in ./amatou_0429: 1124570998357889026_1.jpg
Only in ./amatou_0429: 1138393788676681730_1.jpg
Only in ./amatou_0429: 1138393788676681730_2.jpg
Only in ./amatou_0429: 1138393788676681730_3.jpg
Only in ./amatou_0429: 1138393788676681730_4.jpg
Only in ./amatou_0429: 888150606564675584_1.jpg
Only in ./amatou_0429: 901960030718533633_1.jpg
Only in ./amatou_0429: 901960030718533633_2.jpg
Only in ./amatou_0429: 901960030718533633_3.jpg
Only in ./amatou_0429: 901960030718533633_4.jpg
Only in ./amatou_0429: 906008993302102018_1.jpg
Only in ./amatou_0429: 924062887127085056_1.jpg
Only in ./amatou_0429: 924062887127085056_2.jpg
Only in ./amatou_0429: 924062887127085056_3.jpg
Only in ./amatou_0429: 924062887127085056_4.jpg
Only in ./amatou_0429: 934362369987297280_1.jpg
Only in ./amatou_0429: 940847662462590976_1.jpg
Only in ./amatou_0429: 947335358089281536_1.jpg
Only in ./amatou_0429: 955984561250299904_1.jpg
Only in ./amatou_0429: 955984561250299904_2.jpg
Only in ./amatou_0429: 955984561250299904_3.jpg
Only in ./amatou_0429: 955984561250299904_4.jpg
Only in ./amatou_0429: 959722737722843136_1.jpg
Only in ./amatou_0429: 961560012966395906_1.jpg
Only in ./amatou_0429: 964276358124744704_1.jpg
Only in ./amatou_0429: 968412534934876160_1.jpg
Only in ./amatou_0429: 970625587860549632_1.jpg
Only in ./amatou_0429: 980449686304702466_1.jpg

This is such a mess, some are only in the bare link, some only in the media link, not even sure wtf is happening.

ATM my roundabout way is of course, doing this twice, once for each and then putting them together, but this is really annoying, As i don't seem to get the all results either way, except with both.

Any idea why this happens?

[pxssy]

Additionally, is it possible that retweet images/videos from a particular user be placed separately into a directory under the retweeting user? If so, how can i accomplish that?

As for my other request, how do you use rename tags?

[mikf]

https://twitter.com/amatou_0429 and https://twitter.com/amatou_0429/media are two different timelines on Twitter's website. The first is your regular Twitter timeline with text-only tweets and retweets, and the second on is for media tweets only (images and videos). You can see this for yourself by visiting those two URLs in your browser and scrolling through their tweets (That's essentially what gallery-dl is doing in the background).

As for why the results of those two are so vastly different, I couldn't tell. That's just what Twitter "feels" like presenting when a user pages through those timelines.

ATM my roundabout way is of course, doing this twice, once for each and then putting them together, but this is really annoying, As i don't seem to get the all results either way, except with both.

You can set the directory format strings (in you config file) for twitter in such a way that both normal and media timeline results get written to the same directory:

{
    "extractor":
    {
        "twitter":
        {
            "directory": ["{category}", "{user}"]
        }
    }
}

Additionally, is it possible that retweet images/videos from a particular user be placed separately into a directory under the retweeting user? If so, how can i accomplish that?

This can be done by filtering all non-retweets and, again, setting the directory format string appropriately:

$ gallery-dl --filter 'retweet_id'-o directory='["{category}", "{user}", "Retweets"]' -o retweets=true https://twitter.com/amatou_0429

You could also put all of this in a separate config file and load it only for downloading retweets:

retweets.conf
-------------
{
    "extractor":
    {
        "twitter":
        {
            "image-filter": "retweet_id",
            "directory": ["{category}", "{user}", "Retweets"],
            "retweets": true
        }
    }
}

-------------
$ gallery-dl -c retweets.conf https://twitter.com/amatou_0429

As for my other request, how do you use rename tags?

It's explained here, and here is another example.

[indrakaw]

A note: username/media shows only the media (including replies) of username, it won't show retweet. When the username/ shows retweet and non-replies tweet.

Aside these two, there is username/with_replies with implies username/ but with replies: a full user timeline.

[mikf]

This is fixed with 3346f58 and all the changes/improvements following that.