I wrote a little Script using Python and Tweepy to save the tweets for a list of users and also to get some basic properties for those accounts.
Somehow the number of tweets stated in the user profile under statuses_count
(for an example of the json description of an account:
https://api.twitter.com/1/users/show.json?screen_name=TwitterAPI&include_entities=true )
does not match the number of tweets i get when iterating through the tweets of the same users profile.
I am aware of the fact, that twitter limits the number of tweets per user available through the API to 3200 and even does not guarantee this number, but this behavior does even occur with users who have well less than 3200 tweets
My question is, whether this difference is common and why this happens?
Is this just an issue of the twitter API, is it caused by deleted tweets (maybe they still count for statuses_count but can not be fetched anymore?), ...?
Thanks!
Thomas
I haven't messed with the Twitter API in several months, but I remember back when I was working with it I found inconsistencies due to retweets not showing up when iterating tweets, but getting counted in the number of Tweets. This seems to corroborate that, but its several months old and things may have changed since then.
Make sure include_rts is set to true, t, or 1 (in addition to specifying the same for include_entities, which you have done). When these aren't included by default (e.g. user lists) then you can get fewer tweets than what you specified with count.
The Twitter API documentation isn't clear on what the defaults are so it's safer to explicitly specify these optional parameters. And since you're specifically working with the user timeline you might also want exclude_replies turned off.
Related
I am using Mailchimp's archive URL in PHP -- I am simply fetching the URL and displaying it as it sits, in order to white lable the funky URL IE
https://us17.campaign-archive.com/home/?u=xxxxxyyyyyxxxxyyyy&id=xxxxyyyyyxxxxyy
In doing so I have read through both the Archive and API documentation, and have found nothing on the parameter for row count. It defaults to 20 as stated in the Archive docs, but I know I have seen archives with a larger row count than that. Is anyone familiar enough with the URL parameters used by MailChimp to increase the row count, to say, 100? IE
https://us17.campaign-archive.com/home/?u=xxx&id=yyy&count=100
It's been a problem for years. Even in 2022 there is still no known way for an end-user to get more than past 20 issues from mailchimp, they simply refuse to add/allow that ability.
However the newsletter creator can go into their backend and generate/enable a javascript API that has the &show= parameter, which can be increased.
https://mailchimp.com/help/add-an-email-campaign-archive-to-your-website/
Again, only the campaign creator can do this, not some random end-user/reader.
I'm using the twitter API and would like to get all tweets from a given time period where a given user was mentioned.
I tried the search API with the until parameter where I could get the tweets for the previous 7 days from the date which is not the best, but fine. However, in this case the API only returns the number of tweets without the details.
Any ideas how to do it or what I'm doing wrong?
Thanks!
Strange, https://dev.twitter.com/rest/reference/get/search/tweets says it should return tweet objects. Are you sure you implemented it correctly? Which library are you using? Maybe there is a fault in its implementation. Another library could therefore be the solution.
Don't know whether this satisfies your needs, but you could also try to use the GET statuses/mentions_timeline API . This API returns up to 800 tweets in which the user was mentioned.
EDIT, based on the comments on this post:
You are doing 2 things wrong, namely:
As stated in the documentation the until parameter has a 7 day limit. So you can only retrieve up to 7 days of tweets.
Furthermore, you are making incorrect use of the since parameter. The since parameter specifies since this tweet (so the id of the tweet), not since this date.
Please read the documentation carefully. It is written very precisely and contains all the answers you probably need.
I've noticed that me/home in the Graph API doesn't return posts by certain users. I've tried this in my app as well as just using Graph Explorer. It returns most posts, consistently fails to include posts by certain friends. I don't think this is a caching issue, because I've tried over a period of one day with same results. It's not random either. It's the same handful of posts that are always missing.
I checked the posts in question and don't see anything special. And it's happening with newer posts also.
Do I need to add any special parameters to my request ?
Users, in their Facebook privacy preferences, can configure what data 3rd party applications can see about themselves.
What you are seeing is that certain users have made their activities not viewable to your application.
With https://dev.twitter.com/docs/api/1/get/statuses/user_timeline I can get 3,200 most recent tweets. However, certain sites like http://www.mytweet16.com/ seems to bypass the limit, and my browse through the API documentation could not find anything.
How do they do it, or is there another API that doesn't have the limit?
You can use twitter search page to bypass 3,200 limit. However you have to scroll down many times in the search results page. For example, I searched tweets from #beyinsiz_adam. This is the link of search results:
https://twitter.com/search?q=from%3Abeyinsiz_adam&src=typd&f=realtime
Now in order to scroll down many times, you can use the following javascript code.
var myVar=setInterval(function(){myTimer()},1000);
function myTimer() {
window.scrollTo(0,document.body.scrollHeight);
}
Just run it in the FireBug console. And wait some time to load all tweets.
The only way to see more is to start saving them before the user's tweet count hits 3200. Services which show more than 3200 tweets have saved them in their own dbs. There's currently no way to get more than that through any Twitter API.
http://www.quora.com/Is-there-a-way-to-get-more-than-3200-tweets-from-a-twitter-user-using-Twitters-API-or-scraping
https://dev.twitter.com/discussions/276
Note from that second link: "…the 3,200 limit is for browsing the timeline only. Tweets can always be requested by their ID using the GET statuses/show/:id method."
I've been in this (Twitter) industry for a long time and witnessed lots of changes in Twitter API and documentation. I would like to clarify one thing to you. There is no way to surpass 3200 tweets limit. Twitter doesn't provide this data even in its new premium API.
The only way someone can surpass this limit is by saving the tweets of an individual Twitter user.
There are tools available which claim to have a wide database and provide more than 3200 tweets. Few of them are followersanalysis.com, keyhole.co which I know of.
You can use a tool I wrote that bypasses the limit.
It saves the Tweets in a JSON format.
https://github.com/pauldotknopf/twitter-dump
You can use a Python library snscrape to do it. Or you can use ExportData tool to get all tweets for the user, which returns already preprocessed CSV and spreadsheet files. The first option is free, but has less information and requires more manual work.
Let's say user A follows user B, and B follows A. I want to know the exact date A started following B and viceversa.
Is this information stored on twitter? Can I retrieve it using the API?
To clear out: The point of this question is finding a way to know who followed who first.
(I'm assuming both A and B deleted the notification e-mails)
No Ignacio, you can't. You just can know who follows who but not the date the follow started.
Looking at the API, there's is no way, there are two calls to get the followers:
User Methods/statuses/followers
and
Social Graph Methods/followers/ids
Neither of them returns dates or even a serial that would let you see who started following first. Really, there's no indication that twitter is internally storing this information, neither in the API nor Twitter's web interface.
This is a very old question, but perhaps some might be interested to know that while you cannot get the date at which someone started following, you can at least infer an "earliest possible following date" from the fact that the list of followers is ordered according to date, and the fact that follower objects come with a created_at timestamp.
Here's a Python function for calculating an "earliest possible following date": https://github.com/BernhardClemm/twitter-follow-dates
Of course Twitter stores it, because Twitter sorts followers and following lists by the date ;)
It is possible to do this, but impractical. When you call the followers API you can page the results. Each returned object contains next_cursor and prev_cursor items. These refer to the first and last records in the next and previous pages. These values are time based and can be used to calculate the time that the respective users followed you.
It follows that, if you set the page size to 1, you can walk through the list of follower IDs one at a time and the next_cursor value will allow you to derive the follow time for the next record.
This is reasonably simple to implement, however, in practice, you'll very quickly hit Twitter's API rate limit.