I am trying to get as much data as a I can out of the Twitter API for an academic research project. Even though I only have access to the Standard API the data should be as accurate as possible. I am building myself a "wrapper" around Twarc and other utilities in Python that gets me most of the data I want in just the format I need. A big problem was getting all the replies, but I was able to solve it with a bit of trickery: Searching from the tweet in question onwards and then checking if the tweets in the obtained sample have the original tweet ID in "in_reply_to_tweet_id". Rinse and repeat with those newly obtained tweets.
Then I noticed the new moderation feature Twitter implemented in March. Now the moderated comments under "More replies" do not show up in my search output.
Example: https://twitter.com/NDRreporter/status/1113353224730365952
I find all replies except the following: Under "More replies" ("Mehr Antworten" in German), there is a reply chain started by a extreme right leaning (possibly troll) account ("#Der Steuerzahler") that got moderated and shoved down there. This does not show up in API searches, even if I let the code iterate for over an hour just looking for replies to this particular original tweet.
My question is pretty general: Aside from getting replies as they come in (i.e. before they are moderated) via Filter API, is it possible to find these moderated tweets via the Standard Search API? Not looking for a ready-made solution, general pointers suffice. If I can't find them via Search, then I obviously won't try it with that anymore.
Thanks in advance.
Related
I am trying to understand if there is an option to find out through the API if a tweet is a thread or not, it doesn't matter which API, search for tweets, get tweets, get bookmarks, any of them.
After searching through the API for hours all I see is there is a conversation ID but without getting for each tweet all the replies there is no way to know if it's a thread or not. I also tried to use edit_controls but that also led me no where.
Did someone find a way? Am I missing something? I noticed some people were able to do it so I am guessing it's possible.
Thanks.
Unfortunately, such a feature is not available at the moment and you have a dilemma, the first way is to get the last tweet of a string of tweets using API, which will chain to the main tweet, like the following output (lines 24 to 28):
But the second way is to check the comments of a tweet, and if the user-id of the commenting user is the author of the tweet, it means that you can use this code to receive the comments of a tweet:
for tweet in tweepy.Cursor(
api.search_tweets,
q="to:ZarchiMohammad",
result_type="recent").items(200):
if hasattr(tweet, "in_reply_to_status_id_str"):
if tweet.in_reply_to_status_id_str == tweet_id:
print(f"https://twitter.com/{tweet.user.screen_name}/status/{tweet.id}")
I'm trying to generate multiple related hashtags based on the keyword entered by the user. For example: if the user has typed 'meme', it would generate multiples hashtags related to it (memes, funny memes)
I looked for an Instagram API but didn't find any such endpoints. There are lots of sites and apps available over this thing. Can anyone suggest to me how this would work? I'm not expecting you to do work for me. Just need your guidance on this? Is there any way to achieve this?
https://developers.facebook.com/docs/instagram-api/guides/hashtag-search/ (I goes over all the endpoint but didn't find it, is there any other way to achieve it?)
Demo:- https://toolzu.com/hashtag-generator-for-instagram/ (this brings related hashtags, difficulty and number of times each hashtag has been used)
This api returns hashtags related to a keyword. It's easy to use with axios. Downside is a the limited number of free calls
hashtag api: https://rapidapi.com/miguel.aka.kelter/api/hashtagy-generate-hashtags/
Good luck
I used to use google finance to create portfolios/change them and then display them on my site but since its being removed I'm wondering if there's any good free alternatives?
Basically I have a program that creates different portfolios based on different factors(20 right now), so each of the 20 links on my site direct people to a page that displays the portfolios. I am looking for something that I can use to automatically update the portfolios.
If it helps, my site is basically a free tutorial site that helps people learn how to manage their own portfolios. There's different lessons and then using market data & news(which I already get) I automatically generate a sample portfolio to show them how everything comes together. I liked google finance because they could see all of google's data but they could also click around and dig deeper if they want.
Is there anything I can use to get this result?
I'm not sure I understand exactly what you want, but it sounds like #DirkEddelbuettel's BeanCounter will do what you need.
Or, if you're just looking for quotes see http://www.gummy-stuff.org/Yahoo-data.htm and http://developer.yahoo.com/yql/
With https://dev.twitter.com/docs/api/1/get/statuses/user_timeline I can get 3,200 most recent tweets. However, certain sites like http://www.mytweet16.com/ seems to bypass the limit, and my browse through the API documentation could not find anything.
How do they do it, or is there another API that doesn't have the limit?
You can use twitter search page to bypass 3,200 limit. However you have to scroll down many times in the search results page. For example, I searched tweets from #beyinsiz_adam. This is the link of search results:
https://twitter.com/search?q=from%3Abeyinsiz_adam&src=typd&f=realtime
Now in order to scroll down many times, you can use the following javascript code.
var myVar=setInterval(function(){myTimer()},1000);
function myTimer() {
window.scrollTo(0,document.body.scrollHeight);
}
Just run it in the FireBug console. And wait some time to load all tweets.
The only way to see more is to start saving them before the user's tweet count hits 3200. Services which show more than 3200 tweets have saved them in their own dbs. There's currently no way to get more than that through any Twitter API.
http://www.quora.com/Is-there-a-way-to-get-more-than-3200-tweets-from-a-twitter-user-using-Twitters-API-or-scraping
https://dev.twitter.com/discussions/276
Note from that second link: "…the 3,200 limit is for browsing the timeline only. Tweets can always be requested by their ID using the GET statuses/show/:id method."
I've been in this (Twitter) industry for a long time and witnessed lots of changes in Twitter API and documentation. I would like to clarify one thing to you. There is no way to surpass 3200 tweets limit. Twitter doesn't provide this data even in its new premium API.
The only way someone can surpass this limit is by saving the tweets of an individual Twitter user.
There are tools available which claim to have a wide database and provide more than 3200 tweets. Few of them are followersanalysis.com, keyhole.co which I know of.
You can use a tool I wrote that bypasses the limit.
It saves the Tweets in a JSON format.
https://github.com/pauldotknopf/twitter-dump
You can use a Python library snscrape to do it. Or you can use ExportData tool to get all tweets for the user, which returns already preprocessed CSV and spreadsheet files. The first option is free, but has less information and requires more manual work.
Can someone explain to me in, in REST terms, Twitter's design decision with the parameter placement in these two calls? It seems that the :id placement is inconsistent and arbitrary (although clearly this was deliberate).
GET statuses/:id/retweeted_by
Show user objects of up to 100 members who retweeted the status.
GET statuses/retweets/:id
Returns up to 100 of the first retweets of a given tweet.
There are other similar examples throughout their API (https://dev.twitter.com/docs/api), so I'm definitely missing something.
Thanks!
Just making guesses here
Someone at Twitter once pointed out that the Twitter API runs on several servlets. I can only assume that this was related - it's easier to map /retweets/* than to map every single combination.
Update: I think that the history of the API itself can also be relevant. Twitter's API hasn't really changed much over the past years, and if it did change then it would be because new features would be added. An endpoint like GET statuses/show/:id is old, while GET statuses/retweets/:id is newer. If Twitter at some point decided to change naming conventions, they couldn't just rename the old ones, since it would break applications.
Another theory of mine is that GET statuses/retweets/:id actually doesn't refer to the Tweet :id itself, but is about the tweets that were based on it. GET statuses/:id/retweeted_by is directly related to the tweet itself, by returning users and not other statuses.
I too am often puzzled by the naming consistency. I'm sure they have their reasons though.
I ended up checking with a friend at Twitter, who says:
"I just talked with the guy who originally wrote those two API
endpoints, and he doesn't remember why. To answer your question,
though, there probably isn't a good RESTful reason for that design."