I am getting this error whenever I try to follow someone on Instagram via API no matter how many follows have been done before:
{"meta":{"error_type":"APIError","code":400,"error_message":"Client request limit reached"}}
My app allows authenticated users to follow interesting people. I know that there is a 5000 call/hour limit per authenticated user, but it fails even with new users.
Do my app is reaching some kind of client level limit?
APIs like follow, unfollow, comment are limited to 350 requests per hour. However sending requests from client side will fix this problem to some extent but it allows the users to see your API token.
In this case it looks like it would be beneficial to get some more data from your users. You could use Google analytics to track the "follow" action
https://developers.google.com/analytics/devguides/collection/gajs/eventTrackerGuide
This would give you a timestamp and information about user behavior.
Even with an advertised rate of X requests per hour, one user hammering the service with your API key can cause everyone to get throttled. (Not guaranteed, but pretty common practice for companies to keep their services alive)
It might be a good idea to reset your API, its possible (though unlikely) that someone has acquired your key and is using it.
Related
I am trying to find some specific details on the Github API rate limit.
I have seen many different places suggesting 5000 requests per hour, but I have been limited to 30 according to my response headers.
I have looked at a couple other questions related to github rate limits but haven't found any specifics on them.
I authenticate my requests with OAUTH in the headers.
I recieve Error: Request failed with status code 403 which according to the docs means I'm authenticated.
What I would like to know is:
What are the different rate limits?
Why would I receive 30/hour while someone else may have 5000/hour?
How do I increase it?
You are likely seeing a reduced rate limit because you are not successfully authenticating. You may be providing credentials, but those credentials are not correct, don't have the right scopes, or are otherwise not valid for the resources you're requesting. If you're accessing public information, then you'd get access to that data anyway even without authenticating, so your invalid credentials wouldn't matter but would result in a reduced rate limit.
The reason GitHub provides different rate limits for authenticated and unauthenticated users is to prevent abuse and make it possible to contact users who are engaging in excessive or unacceptable behaviors. For example, if a bot account or OAuth app is causing problems, the maintainer of that service can be contacted, and in an emergency, that account can be prevented from accessing the API until it can be fixed so as not to impact other users.
You can, as the documentation mentions, get an increased rate limit for the API by successfully authenticating. You haven't provided any details about what you're requesting or what scopes you've asked for, so we can't help there; you'll probably want to ask another question with more details. Beyond the limit of 5000 requests, you'd have to ask GitHub Support, although that is a relatively generous limit.
I have successfully set up a DM bot with the Account Activity API. Everything works very well, except that sometimes the message sent to the bot (through the Twitter's web interface or mobile application) doesn’t fire a webhook to my server. The messages could be quick replies responses or plain text.
The reason is obviously not a downtime of my server since I tried to make a conversation between 2 webhook registered users (so my server receive the webhooks for both users) and for the same message sent, I have successfully received the webhook of the sender (the user) but not for the recipient (the bot).
As the bot isn’t in production yet, the reason is not an overload of messages. There is currently only 2 users that make conversations. From my experience, around 10% of messages are "lost".
I'm using the free (sandbox) Account Activity API tier, but as I understand the only differences between the free and paid versions are a higher number of subscriptions (I'm fine with 15) and the “Retries” feature. Regarding this feature, it is specified that “The Account Activity API provides a retry feature when the client’s web app does not return a ‘success’ 200 response for an account activity webhook event.”
It clearly states that the event failure concerns the client’s side, not the Twitter side. Considering this issue (my server doesn't receive the webhook at all), there is no guarantee that every event will be delivered even if in a paid plan.
This is a big inconvenience for bots since a button can only be clicked once, so the user must retry the conversation from the beginning (besides the fact that the bot "doesn't work"...)
So my questions are :
Is anyone here experience this issue ?
Is this a “bug or a feature” of the free Account Activity API ? I mean, at random the free tier doesn't fire the webhook on purpose (even if it's not specified in the docs) ?
Is there a way to see or measure the webhook failures Twitter side, via the dashboard for instance ?
A guess is that the events could be more accurate if the account is verified (with a blue badge) or hit a followers number threshold ? The treatment could be different due to the potential surge of events, so they are monitored with more ressources, thus more reliable ?
I already create a topic in the official Twitter forum and there is at least one other person in the same case, but no official answer from Twitter so far.
Thanks a lot !
BR,
Simon
I've got an official answer from Twitter :
Unfortunately it is not possible to achieve 100% delivery rate when there is only 1 delivery attempt for an event, which is why we have retries (and even then, retries are not a guarantee either). Things can go wrong; maybe internal issues in Twitter Data Centers, routing issues in the internet, hosting issues at your webhook, etc.
So from the time being, it seems that there's not way to have a 100% success delivery when you build a bot on Twitter.
Full answer can be read here.
Apologies if this question has already been asked.
I have followed this guide to create a script that will bulk delete members from a specified Yammer group and I have this working perfectly.
The process used at the minute is obtaining a token for a user, then using that token to remove the user from the group, which takes two API calls.
My question is about "sleeping" to avoid tripping the rate limiter. On the documentation for the Yammer API rate limits (link) it states that "rate limits are per user per app".
In another stackoverflow question (link) it's mentioned that in this context, the user refers to the user token.
Since in my script, I'm only using a single API call per user token (to remove the user from the group), is it necessary implement a sleep to avoid tripping the limit?
I'm also wondering whether the API call to retrieve the token for a user may possibly trip the limiter since it's called using the admin token?
I've run a few tests removing 52 users from a group using a script without any "sleeps" and it completed successfully in around 27 seconds, just trying to understand why this didn't break the limiter.
Thanks in advance!
is it necessary implement a sleep to avoid tripping the limit?
Yes, the admin user (or token if you like) could trip the limit in this case. To be on the safe side, that is, to avoid your app from being (manually or automatically) blocked, you shouldn't make more than request per second to Yammer endpoints that are categorised under "Other Resources". That's the Official guideline.
Well, pretty much what it says on the tin.
I'm really curious about how pages like Statigram do their search functionality without users authentication and not exceeding the limits?
If I'm correct, Instagram API allows 5000 calls per hour, so I believe it's very likely that they indeed have more traffic than 5000 requests per hour.
Maybe It's a dumb question and Statigram has a special deal with Instagram to use their API or maybe they don't use the API and they use some other method?
The only special request you have to send to Instagram is the request to post comments.
The API limit is 5000 requests per hour per access_token or client_id. Every user has their own access_token, so as long as the requests from the third party application uses each individual access token, they will be hard pressed to exceed 5000 per user per hour.
That works out to 83 requests per minute and any user interacting with your application is highly unlikely to hit that.
From the docs:
You are limited to 5000 requests per hour per access_token or client_id overall. Practically, this means you should (when possible) authenticate users so that limits are well outside the reach of a given user.
If you are not using user authentication, you will likely hit the limit with just your client_id.
Most likely they're using one of the following methods:
An arrangement with Instagram
Credential rotation
IP rotation
Heavy caching (especially across credentials or IPs)
Screenscraping
In cases like this, if you don't have a special arrangement, you're almost certainly violating the terms of service. If you think your service is useful enough that Instagram would be willing to whitelist you to make more requests, get in touch with them.
They must have some sort of arrangement with Instagram as #RunscopeAPITools mentions. You are able to post comments to Instagram from Statigram, which requires special permission.
Assume there's a mobile app and a server.
I have question about rate limiting and hoping someone can give some advice on a design as I'm banging my head on how to navigate around rate limit. There must be something I"m missing because the 150 unauthenticated rate limit per IP per hour is extremely low.
Imagine the scenario I want to build is the following (simplified into a trivial example for this discusion). Assume user is signed into Twitter for this entire discussion to remove discussion about oAuth.
Mobile talks to our service to show users twitter friends list. Every time the mobile app is loaded, it will show the entire friends list, and highlighting the new friends that were added within the last 2 days.
That's it. But the trick is that I want to ensure that the friends list is always up to date in the client, which means our server has to have the most recent up to date friends list.
Periodically, I want my server to automatically scan the Twitter friends list for every user of my app to see if new friends have been added.
Our initial design was getting our server to do all the work with this flow:
New User signs in on client, gives access token to server
Server makes call to Twitter REST APIs to get initial friends lists
Server stores the Twitter Friends IDs and shows responds to the client with that list.
Periodically (e.g. every 48 hours), server checks Twitter REST APIs for friends list for each user and compares it to our cached Twitter friends list we have for them to see who is new and to highlight in the mobile app.
The good thing about this is that all the interaction with twitter to get friends list, compare and peridiocally refresh is on the server. Mobile client just makes a single call to my server and gets friends list.
The problem with this design is that it will work for a single user, but since the rate limit is 150 per hour on un-authenticated calls, I will hit my limit as soon as 151 users user my service (which has a fixed IP).
The only solution I can see is to have the client do the work for each user, then send me the friends list which my server caches. This takes care of Step #2 above. However, for Step #4, I'd have to build something into the client to auto refresh twitter friends and send back to the server.
This is super clumsy to have the client involved at all in this Twitter friends list operation.
At first I thought I was crazy and the public unauthenticated APIs like getting friends lists wouldn't be subject to rate limiting. However, according to their docs, it is.
Am I missing something obvious or is the only way to solve this is to put heavy logic into the client?
With whitelisting gone for those that aren't grandfathered or Twitter business partners, I don't think you have any alternative but to have your mobile app do the Twitter API calls from the handset.
Having the handset call Twitter isn't a bad thing by any means. Pretty much every Twitter client in the world does it. One benefit will be that the user will be authenticated to Twitter, and thus her full 350 calls per hour will be available to you. Keep in mind, however, that you should minimize your calls since the user may have other Twitter-aware applications installed on her handset eating into your call allotment, and vice versa.
Now to the solution. The way I would implement your use case would be to first fetch the complete list of friends for your user by calling the friends/ids method.
http://api.twitter.com/1/friends/ids.json?screen_name=yourUsersName
The above call will return the most recent 5,000 friend IDs, in order followed, for #yourUsersName. If you want to fetch more friend IDs than the first 5,000, you'll need to specify the cursor parameter to initiate paging.
Next, I would check the latest list of friends we just fetched against the list on the handset, syncing them by removing any IDs that are no longer present, while adding any that are new.
If we only need the friend IDs, then we're done at a cost of one API call per 5,000 friend IDs. If, however, we need to get user info for these new friends as well, then I would call users/lookup and pass in the list of all new users that we discovered while syncing friend IDs. You can request up to 100 user objects at a time.
http://api.twitter.com/1/users/lookup.json?user_id=123123,5235235,456243,4534563
You user must be authenticated in order to make the above request, but the call can fetch any Twitter user profiles you wish -- not just those that are friends of the authenticated user.
So, let's say for example that a user has 2,500 friends and has never used your app before. In that case, she would burn one call to fetch all of the friend IDs, and 25 calls for her friends' information. That's not too bad to get the app populated with data.
Subsequent calls should be more streamlined with probably only two calls burned (one for the IDs, and one to get the new friends).
Finally, once the data has been updated on the handset, the deltas for the IDs and user data can be gathered up and pushed to your server.
It may even be possible that your server application won't even have to interface with Twitter at all, and that should alleviate the 150 user limit you are encountering.
Some final notes:
Be sure to note in your app's privacy policy that you sync your user's friend list with your server.
I recommend specifying JSON as the return format for all Twitter API calls. It is a much more lightweight document format than XML, and you will typically transfer only about 1/3 to 1/2 as much data over the wire.
Pick a Twitter framework appropriate for your mobile device and your programming language. Twitter access is a commodity these days, and there's little to no reason to reinvent how to access the Twitter API.
I answered a similar question about an approach for efficiently fetching followers here.
Since you are making request on behalf of users you should make those requests be authenticated as those users. Then requests will count against each users own pool of 350 requests/hour.