-
-
Notifications
You must be signed in to change notification settings - Fork 311
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
404 HTTP status code is not handled or not allowed #92
Comments
me too |
me too. |
same here |
same here. |
any luck? |
I also face this issue. |
When trying to access https://twitter.com/search-home, it redirects to https://twitter.com/explore. I think Twitter have removed this functionality altogether. |
We've got https://twitter.com/search-advanced?lang=en-GB. I try to change code but I am new here, so... |
Twitter changed a little bit so we need to alter the url in TweetScraper class into But unfortunately there is still this error @ASIAMI mentioned above, |
Working on it, but still have same error. Must change Xpath - I think it is easy to repair but need someone with experience on it |
I have some experience on Xpath but I am not familiar with how the url should be changed.
I have some time to work on it tomorrow. Can you share me with your progress and problems?
| |
shiwensmile
|
|
[email protected]
|
签名由网易邮箱大师定制
On 09/22/2020 18:10,ASIAMI<[email protected]> wrote:
Working on it, but still have same error. Must change Xpath - I think it is easy to repair but need someone with experience on it
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
I have some experience on Xpath but I am not familiar with how the url should be changed. |
I agree that the url should be changed, but I do not think "https://twitter.com/search/?lang={}".format(lang)` is the right answer, although it's the url displayed when users search the tweet with browser. The right url should response with data containing 'min_position'. |
Hi guys, the way I find out the URL for searching is using Chrome's DevTools to monitor the network activity. Twitter changed the URL for searching and the format of returned data. So you need to modify the code accordingly. Here I give you an example and PRs are welcome. When you search (e.g.,
When you sroll down to the bottom of the page, the same URL is issued again but with a cursor this time, which is used for loading more tweets from the severs:
The response of the URL is like this, where you can find all the tweets and the cursor for the next request:
I'm currently working on some other projects and may update the code later. Hope this can help and PRs are welcome. |
OK, but with this new site - got my response , still -> should I use Bs4 as response is not in JSON Handling different response formats Once you have a response with the desired data, how you extract the desired data from it depends on the type of response:
|
Anyone done with the update? Tried it myself but haven't reached successful one. |
not yet, I've tried with bs4 and got something but still working on it |
@jonbakerfish Can you help me, I can see everything you wrote about. Ale url api.twitter is not allowed. How to get into these data |
Same problem here! |
Hello!
I used Tweetscraper without any problems yesterday, but today always the following issue emerges:
I already changed my IP-Adress via VPN and I changed the user agent, but the issue remains. How am I able to solve this problem? Did twitter change the search url? Trying the url "https://twitter.com/i/search/timeline?l=&f=tweets&q=%23dfl%20since%3A2020-09-16&src=typed&max_position" leads to a 404 response. Thanks for your help!
The text was updated successfully, but these errors were encountered: