-
-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
In light of changes to Twitter's API coming Feb 9 #120
Comments
Hello, I'm using your script for some days and I do agree your statement. |
Saw that earlier, it sucks... Just to confirm, if you get a paid access to the Twitter API, you theoretically still can use it as is right @robertoszek ? Providing you use a valid Twitter token of course. |
Potentially, yes. The thing is nobody really knows how's it going to change or be implemented. |
As a potential replacement, this scrapper seems good, to, and quite light : https://github.com/JustAnotherArchivist/snscrape |
It seems to use the unofficial GraphQL endpoint for scraping data: In addition to We already use pleroma-bot/pleroma_bot/_twitter.py Line 565 in 9a64891
However the I'll look into how feasible would be to use the GraphQL endpoint for our own scraping too. |
Using snscrape, I just did a request to get last 100 tweets for a specific Twitter user (@transportsidf), it worked well. So I don't know what are the limits, but if we can get at least 100 tweets at time, it seems enough for a bot, I think. Gathering tweets... 0
Should I use my API token and it's working fine. I don't know if I do something wrong or if it is a limitation/change in how guest mode works. |
As a bridge solution, maybe pleroma-bot could scrape a Nitter instance? I'd be happy to set up a Nitter instance for my own pleroma-bot to scrape. Then there's zedeus/nitter#389 |
Looks like it's finally here https://tapbots.social/@paul/110109551743991074 |
We just saw our access revoked overnight :/ |
Same here, it finally stopped working yesterday. I’ll need to start experimenting with using RSS via Nitter. Edit: https://github.com/mahrtayyab/tweety looks like a great alternative to use instead of polling RSS. |
My API key switched back to free plan so I can't extract tweets anymore, too. |
I switched to using rsshub, tried nitter but that was very buggy. I think adopting the full graph endpoints would be the best path forward. |
This one, very simple, is working, too : https://gitlab.com/jeancf/twoot |
@robertoszek any chance of future developpments to handle the end of the free API using one of the above solutions ? |
I'm also having problems with twitter api. My bots are no longer working and I can't make it work with RSS source. |
Hello @AltGrCarlos the main problem here is that the creator of this bot is not active in the current time to make the necessary fixes. I would say 75% of the code is still working, and this bot is doing more than a simple scraper : it also updates the user profile, wich is great, and post tweets to mastodon. PS : I don't know about Matrix, in term of live chatting Discord must be more used. |
Hi. I'm not a really good programmer, but I'm trying to understand the code before to make any modification. I'm also trying to develop my own nitter scrapper in order to get the specific information i need from twitter. |
Waiting for a fix to make Pleroma work again, I have set Twoot (previously mentionned) as replacement. It's working fine without API key. |
After last week's API changes breaking nitter, I'm now using https://github.com/12joan/twitter-client to generate RSS for stork. Just so you know stork is still working and still useful. :-) It'd be nice if I could find some way to get profile updates happening while still getting the tweets from RSS; I'll post here again if I figure out a way to do that. |
Hey, sorry for being a lot less active. I've been moving across countries during the last 6 months and between all the logistics and bureaucracy involved (getting a visa, a work permit, finding an apartment, packing, etc.) in addition to keeping a day job, it basically left little to no time to do anything else. I'm glad this project was still somewhat useful for some of you during that time with the scraping functionality implementation still pending. |
Got profile info and pinned tweet gathering working. c96943e These guest accounts seem to be restricted by IP, so only a limited amount can be created from the same host/IP. I'm thinking about adding a flag so they can be created easily on demand: $ pleroma-bot --create-guest-account being dumped to And if you have access to a list of proxies that could be used to generate more accounts at the same time, perhaps passing them as a text file (by a flag or on the config file): $ pleroma-bot --create-guest-account --proxies-file my_proxies.txt And of course the bot would also need to try generating additional guest accounts in the middle of a run if it gets rate limited. |
I have ~50 accounts in my config and run an pleroma-bot every 15 minutes against a nitter rss feed right now as a workaround. The guest accounts last for 30 days and I've ended up needing ~6k guest accounts to keep it running the whole time without erroring out. I use geonode for proxies FYI. |
As long as you use a working Nitter instance as source, you don't have to deal with guest accounts : they are used to scrape Twitter. |
With these changes a lot of nitter instances have either turned off rss or asked people not to scrape them. I run my own so I'm not eating up guest tokens from someone else. Just keep that in mind. Generating guest tokens is extremely cheap on geonode too. |
I guess adding scraping capabilities to the bot has become a priority.
Using RSS feeds as a source will hopefully continue to work after February 9th (if you can find a working Nitter instance, RSSHub or some other third-party site that's still able to generate an RSS feed).
The text was updated successfully, but these errors were encountered: