I'm building a Twitter Application to show specific tweets (that matching pre defined criteria). I used a good library to grab the tweets and before showing them to the user I the tweets must get stored in a local database, so that I have more data and amazing statistics (ego? huh) to be calculated and shown to the user.
The problem is that tweets are not stored in the hashtag, so if I search for the hashtag one week later I will not be able to find the tweets, so I must have a way to show the tweets from the database instead of Twitter API. I decided that I will show data from database when the last tweet from a hashtag (in the database) is stored before than three days or more. when the last tweet is stored in less than three days, then I will ask Twitter to show the tweets.
So I'm asking you if you have an idea how to show tweets from database since my library depends on JSON (or consider it XML). Any ideas?
Store the tweets in CouchDB. If you use twitter streaming api or search api, that should be the most straightforward way for "saving" tweets.
Related
From the picture, you can see how followers statistics looks on minter.io
The only way how I imagine I would count the followers change: I would download the list of all he followers every day by the Instagram API to my DB. And after having this history already can calculate any change.
But on minter.io you can have such a graphics after few minutes after registration... How???
They are probably storing this information on a daily basis and hence are able to keep a historical trend.
If you go to the minter.io website, they mention at the bottom that they have collected data for close to 198 million accounts. I guess you were one of those.
You don't need to get the list of all followers just to show the absolute change in the numbers. The Instagram API gives that directly when you query any of the endpoints giving user information.
I know how it works at smartmetrics.co.
Smartmetrics collects information about all followers of tracked accounts and build history based on this data. So if you followed someone who is already tracked, you can get history for your account.
But minter makes fake linear graph according to some tests: How to Get Historical Data from Instagram API
Crowdbabble and Minter re-use Twitter tokens, which allows them to collect data on millions of accounts. This gives you the historical data that you want -- change in followers over time. As an individual, you aren't able to access the Twitter API and aggregate data like that for storage as easily. You don't have thousands of people giving you tokens that you can then scrape and store on a regular basis.
Crowdbabble has a free 14 day trial with no payment info required. If you don't want in-depth analytics, Twittercounter will give you your follower numbers over the past 30 days -- you can view each day separately.
I wrote a little Script using Python and Tweepy to save the tweets for a list of users and also to get some basic properties for those accounts.
Somehow the number of tweets stated in the user profile under statuses_count
(for an example of the json description of an account:
https://api.twitter.com/1/users/show.json?screen_name=TwitterAPI&include_entities=true )
does not match the number of tweets i get when iterating through the tweets of the same users profile.
I am aware of the fact, that twitter limits the number of tweets per user available through the API to 3200 and even does not guarantee this number, but this behavior does even occur with users who have well less than 3200 tweets
My question is, whether this difference is common and why this happens?
Is this just an issue of the twitter API, is it caused by deleted tweets (maybe they still count for statuses_count but can not be fetched anymore?), ...?
Thanks!
Thomas
I haven't messed with the Twitter API in several months, but I remember back when I was working with it I found inconsistencies due to retweets not showing up when iterating tweets, but getting counted in the number of Tweets. This seems to corroborate that, but its several months old and things may have changed since then.
Make sure include_rts is set to true, t, or 1 (in addition to specifying the same for include_entities, which you have done). When these aren't included by default (e.g. user lists) then you can get fewer tweets than what you specified with count.
The Twitter API documentation isn't clear on what the defaults are so it's safer to explicitly specify these optional parameters. And since you're specifically working with the user timeline you might also want exclude_replies turned off.
With https://dev.twitter.com/docs/api/1/get/statuses/user_timeline I can get 3,200 most recent tweets. However, certain sites like http://www.mytweet16.com/ seems to bypass the limit, and my browse through the API documentation could not find anything.
How do they do it, or is there another API that doesn't have the limit?
You can use twitter search page to bypass 3,200 limit. However you have to scroll down many times in the search results page. For example, I searched tweets from #beyinsiz_adam. This is the link of search results:
https://twitter.com/search?q=from%3Abeyinsiz_adam&src=typd&f=realtime
Now in order to scroll down many times, you can use the following javascript code.
var myVar=setInterval(function(){myTimer()},1000);
function myTimer() {
window.scrollTo(0,document.body.scrollHeight);
}
Just run it in the FireBug console. And wait some time to load all tweets.
The only way to see more is to start saving them before the user's tweet count hits 3200. Services which show more than 3200 tweets have saved them in their own dbs. There's currently no way to get more than that through any Twitter API.
http://www.quora.com/Is-there-a-way-to-get-more-than-3200-tweets-from-a-twitter-user-using-Twitters-API-or-scraping
https://dev.twitter.com/discussions/276
Note from that second link: "…the 3,200 limit is for browsing the timeline only. Tweets can always be requested by their ID using the GET statuses/show/:id method."
I've been in this (Twitter) industry for a long time and witnessed lots of changes in Twitter API and documentation. I would like to clarify one thing to you. There is no way to surpass 3200 tweets limit. Twitter doesn't provide this data even in its new premium API.
The only way someone can surpass this limit is by saving the tweets of an individual Twitter user.
There are tools available which claim to have a wide database and provide more than 3200 tweets. Few of them are followersanalysis.com, keyhole.co which I know of.
You can use a tool I wrote that bypasses the limit.
It saves the Tweets in a JSON format.
https://github.com/pauldotknopf/twitter-dump
You can use a Python library snscrape to do it. Or you can use ExportData tool to get all tweets for the user, which returns already preprocessed CSV and spreadsheet files. The first option is free, but has less information and requires more manual work.
My site needs the Twitter equivalent of the Facebook Comments plugin... You may be aware of the fact that Twitter only searches back about a week, so comments would be lost just a few days after being made! Is there a service that would allow me to show the tweets that contain the given page's unique #hashtag? I'm looking to put together a system that searches for the hashtag and puts those tweets up for display, even if the tweets are old. Does anyone know how to make this possible? I'm not that good of a programmer, FYI...
No. The Twitter Search API will not let you search that far back.
The Search API is not complete index of all Tweets, but instead an index of recent Tweets. At the moment that index includes between 6-9 days of Tweets.
You cannot use the Search API to find Tweets older than about a week.
From https://dev.twitter.com/docs/using-search
After many tests, I've been unable to get the Twitter Search API to return more than 80% of tweets containing a specific keyword or hashtag. This is not related to the maximum number of results, one test involved a hashtag which had been tweeted 50 times and only 15 of them were returned by the Twitter Search API. The same results were returned when using Twitter's own search tool.
Is the Twitter Search API simply a tool for getting estimates and trends, rather than accurate data?
Has anyone found a way to capture 100% of tweets containing a specific keyword or hashtag?
Twitter filters search api for better results. Here is a quote from developer site:
Both the Streaming API and the Search
API filter, and on some end-points,
discard, statuses created by a small
proportion of accounts based upon
status quality metrics. For example,
frequent and repetitious status
updates may, in some instances, and in
combination with other metrics, result
in a different status quality score
for a given account.
Search api simply returns a subset of the found tweets.