From the picture, you can see how followers statistics looks on minter.io
The only way how I imagine I would count the followers change: I would download the list of all he followers every day by the Instagram API to my DB. And after having this history already can calculate any change.
But on minter.io you can have such a graphics after few minutes after registration... How???
They are probably storing this information on a daily basis and hence are able to keep a historical trend.
If you go to the minter.io website, they mention at the bottom that they have collected data for close to 198 million accounts. I guess you were one of those.
You don't need to get the list of all followers just to show the absolute change in the numbers. The Instagram API gives that directly when you query any of the endpoints giving user information.
I know how it works at smartmetrics.co.
Smartmetrics collects information about all followers of tracked accounts and build history based on this data. So if you followed someone who is already tracked, you can get history for your account.
But minter makes fake linear graph according to some tests: How to Get Historical Data from Instagram API
Crowdbabble and Minter re-use Twitter tokens, which allows them to collect data on millions of accounts. This gives you the historical data that you want -- change in followers over time. As an individual, you aren't able to access the Twitter API and aggregate data like that for storage as easily. You don't have thousands of people giving you tokens that you can then scrape and store on a regular basis.
Crowdbabble has a free 14 day trial with no payment info required. If you don't want in-depth analytics, Twittercounter will give you your follower numbers over the past 30 days -- you can view each day separately.
Related
I have a project that involves having public data downloaded from Google plus, can you give me a reference on how I can download like 1 GB of any type of public data from Google plus?
The data can be posts or circles information. I've tried to work with developer tools but the far I got is downloading my own profile information but what I need is public data.
Thanks !
There is no truly "public" data on Google+.
Every stream is unique to a user.
Try viewing the site without logging in, and you'll see what I mean.
Since users have the ability to block other users from viewing even their "public" posts, before Google shows you a post they check to see if you're on the blocked list. For them to be able to do that, you have to be logged in.
Your best bet would be to create a dummy account and only look at your nearby stream or What's Hot.
Otherwise you'd need to circle users, and that would create the stream. G+ is not like twitter. There's no firehose to speak of.
To programmatically cull data, you would have to use their API, but even then their HTTP API limits you to 20 results per search and you have to provide a query.
You could get up to 100 results per user if you picked individuals and got their userids, but again there's not a programmatic way to get a bulk dump.
You could randomly select users by using an activity search for a dictionary entry, and then seed that into the activity listing api... something like (in pure pseudocode)
for Random word in dictionary
group = userids from GET https://www.googleapis.com/plus/v1/activities?query=[word]
for userid in group
GET https://www.googleapis.com/plus/v1/people/[userid]/activities/collection/public
Actual code would of course depend on the language.
I wrote a little Script using Python and Tweepy to save the tweets for a list of users and also to get some basic properties for those accounts.
Somehow the number of tweets stated in the user profile under statuses_count
(for an example of the json description of an account:
https://api.twitter.com/1/users/show.json?screen_name=TwitterAPI&include_entities=true )
does not match the number of tweets i get when iterating through the tweets of the same users profile.
I am aware of the fact, that twitter limits the number of tweets per user available through the API to 3200 and even does not guarantee this number, but this behavior does even occur with users who have well less than 3200 tweets
My question is, whether this difference is common and why this happens?
Is this just an issue of the twitter API, is it caused by deleted tweets (maybe they still count for statuses_count but can not be fetched anymore?), ...?
Thanks!
Thomas
I haven't messed with the Twitter API in several months, but I remember back when I was working with it I found inconsistencies due to retweets not showing up when iterating tweets, but getting counted in the number of Tweets. This seems to corroborate that, but its several months old and things may have changed since then.
Make sure include_rts is set to true, t, or 1 (in addition to specifying the same for include_entities, which you have done). When these aren't included by default (e.g. user lists) then you can get fewer tweets than what you specified with count.
The Twitter API documentation isn't clear on what the defaults are so it's safer to explicitly specify these optional parameters. And since you're specifically working with the user timeline you might also want exclude_replies turned off.
I'm building a Twitter Application to show specific tweets (that matching pre defined criteria). I used a good library to grab the tweets and before showing them to the user I the tweets must get stored in a local database, so that I have more data and amazing statistics (ego? huh) to be calculated and shown to the user.
The problem is that tweets are not stored in the hashtag, so if I search for the hashtag one week later I will not be able to find the tweets, so I must have a way to show the tweets from the database instead of Twitter API. I decided that I will show data from database when the last tweet from a hashtag (in the database) is stored before than three days or more. when the last tweet is stored in less than three days, then I will ask Twitter to show the tweets.
So I'm asking you if you have an idea how to show tweets from database since my library depends on JSON (or consider it XML). Any ideas?
Store the tweets in CouchDB. If you use twitter streaming api or search api, that should be the most straightforward way for "saving" tweets.
After many tests, I've been unable to get the Twitter Search API to return more than 80% of tweets containing a specific keyword or hashtag. This is not related to the maximum number of results, one test involved a hashtag which had been tweeted 50 times and only 15 of them were returned by the Twitter Search API. The same results were returned when using Twitter's own search tool.
Is the Twitter Search API simply a tool for getting estimates and trends, rather than accurate data?
Has anyone found a way to capture 100% of tweets containing a specific keyword or hashtag?
Twitter filters search api for better results. Here is a quote from developer site:
Both the Streaming API and the Search
API filter, and on some end-points,
discard, statuses created by a small
proportion of accounts based upon
status quality metrics. For example,
frequent and repetitious status
updates may, in some instances, and in
combination with other metrics, result
in a different status quality score
for a given account.
Search api simply returns a subset of the found tweets.
Let's say user A follows user B, and B follows A. I want to know the exact date A started following B and viceversa.
Is this information stored on twitter? Can I retrieve it using the API?
To clear out: The point of this question is finding a way to know who followed who first.
(I'm assuming both A and B deleted the notification e-mails)
No Ignacio, you can't. You just can know who follows who but not the date the follow started.
Looking at the API, there's is no way, there are two calls to get the followers:
User Methods/statuses/followers
and
Social Graph Methods/followers/ids
Neither of them returns dates or even a serial that would let you see who started following first. Really, there's no indication that twitter is internally storing this information, neither in the API nor Twitter's web interface.
This is a very old question, but perhaps some might be interested to know that while you cannot get the date at which someone started following, you can at least infer an "earliest possible following date" from the fact that the list of followers is ordered according to date, and the fact that follower objects come with a created_at timestamp.
Here's a Python function for calculating an "earliest possible following date": https://github.com/BernhardClemm/twitter-follow-dates
Of course Twitter stores it, because Twitter sorts followers and following lists by the date ;)
It is possible to do this, but impractical. When you call the followers API you can page the results. Each returned object contains next_cursor and prev_cursor items. These refer to the first and last records in the next and previous pages. These values are time based and can be used to calculate the time that the respective users followed you.
It follows that, if you set the page size to 1, you can walk through the list of follower IDs one at a time and the next_cursor value will allow you to derive the follow time for the next record.
This is reasonably simple to implement, however, in practice, you'll very quickly hit Twitter's API rate limit.