I'm creating an application using the Twitter API and want to get the last Tweet tweeted. Not the last Tweet from people I follow or something like this, the actual last Tweet from all around the world in that moment. Is there a possible way for that?
Not really possible. Best you can do is get the latest tweet from an approximately one percent sampling of all tweets.
I think the only way you'd be able to do that accurately would be to consume the Twitter firehose. As an alternative, you could use the statuses/sample realtime API to get Tweets from 1% of the current volume of the firehose. Another option would be to use statuses/filter with a geofence over the whole planet, but again this would just be a 1% dataset.
Related
I need the historical numbers of followers for a set of 60 Twitter users. Unfortunately the official Twitter API only returns the most current follower count. I know that there is probably no one-size-fit all and fool-proof way to get an accurate follower growth graph for all users in my list. However, would there be a good and logical way to estimate or deduce this based on other information available through Twitter API (e.g. number of retweets, likes)? I read somewhere that retweet rate grows proportionally with number of followers.
Any help or advice would be much appreciated. Thanks!
I think any approximation would be based on some criteria that would most probably not be applicable to all users.
From my point of view, I would start recording the followers from now on in your database and build a graph starting from today and not from the past.
There are services that allow you to buy or access historical data. But only if they have been tracking the account already. For large accounts, it is more likely, but for smaller accounts they probably haven't been tracking them.
https://twittercounter.com/pages/buy-stats/apple
I haven't used these services but I assume they are accurate.
Does anyone know what's the ratio between the number of tweets we get from twitter sample API over the total number of tweets which the Twitter server receives? I am doing some analysis based on the data read from the sample API, and would like to estimate the actual workload handled by Twitter server. I observed that the number of tweets we get from the API varies over time. So, I presume it is something like percentage sample. Any clue is highly appreciated.
Thanks
The sample stream /statuses/sample does return roughly 1% of all tweets. Twitter samples the tweets by delivering only tweets created within a 10-millisecond window out of the 1,000 milliseconds in every second. If you want more details, you can read my blog post: http://blog.falcondai.com/2013/06/666-and-how-twitter-samples-tweets-in.html
When Twitter Spritzer (basically the old-fashioned Streaming API) was launched, it was supposedly about 1-2% of all tweets. Based on my use of the current Streaming API, I'd be surprised if it was any more than 1% right now, and possibly less. According to the docs, the "Twitter streaming volume is not constant," but they neglect to mention if the volume outputted by the API is proportional to the rate of actual tweets.
On 2 February 2015 Twitter announced intent to reset the streaming API sample rate to 1% (it had crept higher unintentionally):
The public Streaming API sample endpoints (aka POST statuses/filter and GET statuses/sample) are intended to be levelled at approximately 1% of the public Tweet volumes at any time.
Due to some past inconsistencies in configuration, there have been periods of time where the volumes of Tweets delivered via the Streaming API may have exceeded these parameters.
This notice is to indicate that over the next couple of weeks, we will be making changes to the public Streaming API to rebalance the volume of Tweets at the 1% capacity that was intended.
This plot shows the effect of the reset on a typical tweet stream.
This is something I found at
https://brightplanet.com/2013/06/25/twitter-firehose-vs-twitter-api-whats-the-difference-and-why-should-you-care/. I hope you find this useful.
Studies have estimated that using Twitter’s Streaming API users can
expect to receive anywhere from 1% of the tweets to over 40% of tweets
in near real-time.
There are references to the studies they have cited at the bottom of the webpage.
Assume you have access to the Twitter firehose, either directly or through a Gnip/Datasift deal. You can analyze who users are following and the tweet streams. The end goal should be to have an ontology of interests, with weighted interests per user.
I am completely stuck even starting to work on this problem. Any guidance is appreciated.
This seems to be along the lines of what you are looking for
http://wthashtag.com/
I'm pretty certain that Twitter has a category listing for lists, at least, in their own API as well.
GL
I'd like to pull all of a user's tweets. I could do this the hard way (manually scraping twitter) or the easy way: using their api. The problem with the easy (api) way is that I seem to be limited to the 200 most recent tweets. What's a simple way to get all tweets?
Thanks
Yes you can get up to 3,200 historical tweets by requesting as follows...
Make a request to:
http://api.twitter.com/1/statuses/user_timeline.format
And use the count parameter 200 and iterate through the page parameter from page 1 to 16 or until there are no more tweets.
Thats the only thing you can currently do because Twitter specifically say they prevent this in their API Doc...
https://apiwiki.twitter.com/Things-Every-Developer-Should-Know#6Therearepaginationlimits
I would add, please don't screen-scrape because it will cause undue load on Twitter and in bulk requests it would probably get your server blocked from accessing Twitter.
you can make sure you get all future tweets by subscribing to your Twitter RSS feed with Google Reader. Then you can use their infinite scrolling feature to look back to the first tweet tracked.
The webapps that depend on the public timeline of twitter, how often do they collect the data? There must be hundreds of thousands of messages every minute, correct? How do they manage to collect all the tweets, without missing any of them?
Some services (Friendfeed is a good example) are granted access to the Twitter Streaming API, aka the 'firehose'. It requires approval and a written agreement.
The publictimeline is not a great place to mine data anymore. Twitter now uses its Streaming APIs to output tweets like crazy. The closest comparison to the publictimeline would be the spritzer method, but that only includes a small sample. If you need to gather all (or more) tweets than the spritzer method, you'll need to sign a written agreement to get access to other Streaming API (HTTP push) feeds, such as the firehose feed, which returns all public tweets.
The twitter API is rate limited, as has been said. The public timeline (twitter.com/public_timeline) is not rate limited in the same sense, but it is only updated every 5 seconds, so most tweets never appear there.
There are I think three or four companies that have access to the firehose, as Twitter's full feed is called. FriendFeed is one of these. Another is Gnip. Gnip resells the feed to other companies. This is probably the only feasible way to get a full twitter feed.
Go here:
http://twitter.com/help/request_whitelisting
and get your account white-listed (allows 20,000 per hour) if 100 requests per hour isn't enough.
#ceejayoz its not 100 GET requests its 100 requests in general excluding a few requests like verify_credentials and rate_limit_status.