I am trying to get as much data as a I can out of the Twitter API for an academic research project. Even though I only have access to the Standard API the data should be as accurate as possible. I am building myself a "wrapper" around Twarc and other utilities in Python that gets me most of the data I want in just the format I need. A big problem was getting all the replies, but I was able to solve it with a bit of trickery: Searching from the tweet in question onwards and then checking if the tweets in the obtained sample have the original tweet ID in "in_reply_to_tweet_id". Rinse and repeat with those newly obtained tweets.
Then I noticed the new moderation feature Twitter implemented in March. Now the moderated comments under "More replies" do not show up in my search output.
Example: https://twitter.com/NDRreporter/status/1113353224730365952
I find all replies except the following: Under "More replies" ("Mehr Antworten" in German), there is a reply chain started by a extreme right leaning (possibly troll) account ("#Der Steuerzahler") that got moderated and shoved down there. This does not show up in API searches, even if I let the code iterate for over an hour just looking for replies to this particular original tweet.
My question is pretty general: Aside from getting replies as they come in (i.e. before they are moderated) via Filter API, is it possible to find these moderated tweets via the Standard Search API? Not looking for a ready-made solution, general pointers suffice. If I can't find them via Search, then I obviously won't try it with that anymore.
Thanks in advance.
I'm looking to somehow tell users to upload a picture and do something, either tag or # our company so we are notified in real time, i just need to fetch the url of the photo that was uploaded and tagged or # or ANYTHING.
the hashtag search doesn't seem to work correctly, i've tried multiple hashtags and they wont retrieve them in any type of real time or at all.
the real-time subscription only notifies me that there is a new photo tagged that i subscribed to, so i have api search for a list of new media with that tag, which goes back to point 1, its not accurate or reliable.
so do i have any options at all on instagram?
If not do I have any options on facebook? any real time keyword search to grab the url of the photo they uploaded with a unique tag.
Check out https://github.com/chapmanu/hummingbird, it might be what you're looking for.
My experience with the Instagram Real-Time API is, that it is rather reliable.
When you get a real-time ping and then call the endpoint for your hashtag, the results are ordered newest first.
You can then paginate until you hit the last known result.
Maybe this tutorial helps: http://www.mtrinitaria.com/mykhel/tutorial-instagram-real-time-photo-update-api-with-php/
Right, this is really working on my nerves, but Instagram has to do something about their bloody documentation.
I am already trying for a week to live update my website with new instagram posts without refreshing the page. Twitter was fairly easy, but instagram is just one big mess. Basically I use the realtime Instagram API, the callback and all that stuff is working fine, but thanks to Instagram it does not return me an ID from the post that is new, the callback only returns some basic stuff:
[{"changed_aspect": "media", "object": "tag", "object_id": "nofilter", "time": 1391091743, "subscription_id": xxxxx, "data": {}}]
with this data you are nothing, except for the Tag, but I knew the tag before this callback too so doesn't matter. It actually only tells me that there is a new post. I have tried doing the same request as when the page loads, when this callback occurs, and get the posts that are newer than those already on the page. Unfortunately I have not succeeded in this yet. I have picked the ID from the last posted instagram post, and checked if it is in the callback request, and it's not.
What am I doing wrong?
I'd appreciate some help, thanks!
Edit:
I'd like to note that this is not only a problem with the realtime api, but also with the normal API. I just don't know how to compare data so I don't get duplicates in my database(normal api), or on my website (realtime). I can't find any tutorial or documentation (Yes, I might be blind), that explains to me how to compare data. I can only find the min_id and max_id, but no explanation what these id's contains. I checked these id's with id's from results, and they do not match. It's not an ID from a media item.
I also checked the next_url, and in my logic thinking, this should be a URL to the next page (like Twitter).
Am I looking at this all wrong?
Ok strike my old answer, I changed the way I do this. Here's how I'll do it now.
I still wait for 10 hits on my Real-time subscription, when I reach 10 I send off a new thread (if one is not already running).
The sync thread queries my DB for a value, I need the last min_tag_id I used. Then I query:
https://api.instagram.com/v1/tags/*/media/recent?access_token=*&min_tag_id=*
Try it out here: https://api.instagram.com/v1/tags/montreal/media/recent?access_token=*
You'll get 20 results, and a min_tag_id value. Append that to your url, you'll see you get no results. Wait a couple of seconds and refresh. Eventually you'll get some media, and a new min_tag_id.
(You can ignore the "next_url" value they give you, you won't be using that).
Basically you only need to store that min_tag_id and query until you have no more results, that means you're done then.
When you get a subscription push, you need to query that endpoint (tag / recent).
I normally start an synchronous thread to perform this so I can answer in under 2 seconds to Instagram.
Then you parse that endpoint and look for a "next url" value.
Keep querying that end point, parsing the media and going to the next url until you find your stop condition.
For me I try to match 10 consecutive records in my DB. Basically from the tag, I store media when then meet my business rules.
The Instagram documentation is accurate and actually well written.
The realtime API is working correctly. As stated in the documentation:
The changed data is not included in the payload, so it is up to you
how you'd like to fetch the new data. For example, you may decide only
to fetch new data for specific users, or after a certain number of
photos have been posted.
http://instagram.com/developer/realtime/
You only receive a notification that an update has happened to your subscribed object. It is up to you to call the API to find out what that data is.
You can call the /tags/[tag-name]/media/recent with an access token that you have previously stored on your own server or DB. Then, you should be able to compare the data returned from that endpoint with any data you have retrieved prior, and just pull the objects that you do not yet have.
We are working on some analytics using the amount a user is retweeted or mentioned... I can't seem to find a way to get these numbers using the apis does anyone have any ideas?
https://api.twitter.com/1/statuses/user_timeline.json?include_entities=true&include_rts=true&screen_name={screen_name}&count={count}
it's important to include the line include_entities=true to the request. This will give you an expanded response including re-tweet and mention counts.
Get Status / User Timeline
Twitter API Console
Update:
to get tweets from the last 90 days, there is a Node.js library you can use called Snapbird
https://github.com/remy/snapbird
.. and here is another resource covering the same topic.
http://blog.tweetsmarter.com/twitter-search/10-ways-and-20-features-for-searching-old-tweets/
For some reason, when searching on one specific Twitter user, the search API return nothing. (ie http://search.twitter.com/search.atom?q=+from%3ATWITTERHANDLE_A) TWITTERHANDLE_A here is the Twitter account name. This user has been active for over a month, has had many RTs, #s and has sent such tweets out as well.
Meanwhile, I created a new Twitter account - we will call it TWITTERHANDLE_B. Immediately after I created the account, I sent 1 tweet and performed the same search as above (http://search.twitter.com/search.atom?q=+from%3ATWITTERHANDLE_B)
The tweet was returned.
Is there ANYWAY to find out if and/or why a particular user would be blocked from search results? Thanks so much for any help... I'm going crazy here. Twitter's documentation just says sometimes a user's tweets won't be searched!
The documentation you refer to has a link to this page:
http://support.twitter.com/forums/10713/entries/42646
The bottom entry addresses your issue, hopefully you can fix it that way.
Bear in mind the search API only grabs results from the last couple of weeks, so if the user hasn't tweeted in a while then there won't be any results.
You could try not using the Search API and grab the results directly:
http://twitter.com/statuses/user_timeline/-username-.json
Works fine for http requests at least... >_<