With https://dev.twitter.com/docs/api/1/get/statuses/user_timeline I can get 3,200 most recent tweets. However, certain sites like http://www.mytweet16.com/ seems to bypass the limit, and my browse through the API documentation could not find anything.
How do they do it, or is there another API that doesn't have the limit?
You can use twitter search page to bypass 3,200 limit. However you have to scroll down many times in the search results page. For example, I searched tweets from #beyinsiz_adam. This is the link of search results:
https://twitter.com/search?q=from%3Abeyinsiz_adam&src=typd&f=realtime
Now in order to scroll down many times, you can use the following javascript code.
var myVar=setInterval(function(){myTimer()},1000);
function myTimer() {
window.scrollTo(0,document.body.scrollHeight);
}
Just run it in the FireBug console. And wait some time to load all tweets.
The only way to see more is to start saving them before the user's tweet count hits 3200. Services which show more than 3200 tweets have saved them in their own dbs. There's currently no way to get more than that through any Twitter API.
http://www.quora.com/Is-there-a-way-to-get-more-than-3200-tweets-from-a-twitter-user-using-Twitters-API-or-scraping
https://dev.twitter.com/discussions/276
Note from that second link: "…the 3,200 limit is for browsing the timeline only. Tweets can always be requested by their ID using the GET statuses/show/:id method."
I've been in this (Twitter) industry for a long time and witnessed lots of changes in Twitter API and documentation. I would like to clarify one thing to you. There is no way to surpass 3200 tweets limit. Twitter doesn't provide this data even in its new premium API.
The only way someone can surpass this limit is by saving the tweets of an individual Twitter user.
There are tools available which claim to have a wide database and provide more than 3200 tweets. Few of them are followersanalysis.com, keyhole.co which I know of.
You can use a tool I wrote that bypasses the limit.
It saves the Tweets in a JSON format.
https://github.com/pauldotknopf/twitter-dump
You can use a Python library snscrape to do it. Or you can use ExportData tool to get all tweets for the user, which returns already preprocessed CSV and spreadsheet files. The first option is free, but has less information and requires more manual work.
Related
Context
I am in the process of providing some consultancy on doing a HTTP GET using YouTube Data API V3; in order to develop a Windows based application to GET a list of results from Youtube, for say a specific CATEGORY, or a specific TAG.
We are open to using any programming language(I'm from a C++ background and am hoping You tube will support direct HTTP connections without using Google client SDK and so on) to connect to YouTube and (HTTP) GET data.(Once a month or so, so YouTube API quotas should not be problem).
The Issue
We are being told by some of my client's web developers that YouTube API v3 will only return a maximum of 500 records/results, for say a query that returns JUST the Total viewers, the Video's link, and basic meta data such as that.
S, say I wish to find 5,000 results for category "House music" or "basketball" - and I have the Developer Key etc are all set up, would that be possible?
If so, what GET fields would I need to populate(such as "max_results_per_page")?
Thank you.
The API won't provide more than ~500 search results for any arbitrary query. It's by design. Technically, it means that the nextPageToken field won't be returned once you hit ~500 results. No additional parameter can change that.
If you want more than ~500 results for a query, you have to split it into more specific sub-queries. I'd suggest using the publishedAfter and publishedBefore parameters to achieve that, but feel free to experiment with the other ones here.
This only holds for the search-Query. Other queries like "PlaylisItem:list" deliver more results. I have tested with 100.000 items to get the videos of a playlist.
I wrote a little Script using Python and Tweepy to save the tweets for a list of users and also to get some basic properties for those accounts.
Somehow the number of tweets stated in the user profile under statuses_count
(for an example of the json description of an account:
https://api.twitter.com/1/users/show.json?screen_name=TwitterAPI&include_entities=true )
does not match the number of tweets i get when iterating through the tweets of the same users profile.
I am aware of the fact, that twitter limits the number of tweets per user available through the API to 3200 and even does not guarantee this number, but this behavior does even occur with users who have well less than 3200 tweets
My question is, whether this difference is common and why this happens?
Is this just an issue of the twitter API, is it caused by deleted tweets (maybe they still count for statuses_count but can not be fetched anymore?), ...?
Thanks!
Thomas
I haven't messed with the Twitter API in several months, but I remember back when I was working with it I found inconsistencies due to retweets not showing up when iterating tweets, but getting counted in the number of Tweets. This seems to corroborate that, but its several months old and things may have changed since then.
Make sure include_rts is set to true, t, or 1 (in addition to specifying the same for include_entities, which you have done). When these aren't included by default (e.g. user lists) then you can get fewer tweets than what you specified with count.
The Twitter API documentation isn't clear on what the defaults are so it's safer to explicitly specify these optional parameters. And since you're specifically working with the user timeline you might also want exclude_replies turned off.
Is there a way to use the Twitter API to get a user's tweets for a specific time range?
It doesn't seem to be in statuses/user_timeline.
You can use the since and until operator.
https://api.twitter.com/1.1/statuses/user_timeline.json?include_entities=true&include_rts=true&screen_name=your_screen_name&since:2011-05-16&until:2011-08-16
But it will do you little good because tweets are searchable in a small time window. To get around this check out several resources,
Snapbird - https://github.com/remy/snapbird
and
20 ways of searching old tweets
My site needs the Twitter equivalent of the Facebook Comments plugin... You may be aware of the fact that Twitter only searches back about a week, so comments would be lost just a few days after being made! Is there a service that would allow me to show the tweets that contain the given page's unique #hashtag? I'm looking to put together a system that searches for the hashtag and puts those tweets up for display, even if the tweets are old. Does anyone know how to make this possible? I'm not that good of a programmer, FYI...
No. The Twitter Search API will not let you search that far back.
The Search API is not complete index of all Tweets, but instead an index of recent Tweets. At the moment that index includes between 6-9 days of Tweets.
You cannot use the Search API to find Tweets older than about a week.
From https://dev.twitter.com/docs/using-search
We are working on some analytics using the amount a user is retweeted or mentioned... I can't seem to find a way to get these numbers using the apis does anyone have any ideas?
https://api.twitter.com/1/statuses/user_timeline.json?include_entities=true&include_rts=true&screen_name={screen_name}&count={count}
it's important to include the line include_entities=true to the request. This will give you an expanded response including re-tweet and mention counts.
Get Status / User Timeline
Twitter API Console
Update:
to get tweets from the last 90 days, there is a Node.js library you can use called Snapbird
https://github.com/remy/snapbird
.. and here is another resource covering the same topic.
http://blog.tweetsmarter.com/twitter-search/10-ways-and-20-features-for-searching-old-tweets/