Imagine an API with an endpoint called /valuableData. The server clearly states that the limits for the API are 500 requests/s in total and 20 requests/s per person. This server does not check your IP so I could make 500 request/s with 500 good working proxies without the API going offline. Now imagine that person A sends 1000 requests/s and person B sends 20 requests/s which causes the 500 limit to be hit and the API goes offline. As soon as the API comes back online, how many valid answers does person A get back per second and how many person B? How does the server/API handle this? Does person A get back 480 valid answers per second and person B 20? Or does person A get all the max of 500 answers/s?
This depends on the protocol.
HTTP which normally is used for REST is in itself stateless and client-initiated and therefore falls short of the requirements.
Your scenario would require a protocol which:
Knows who the clients are/were
Can push responses after coming online again without client involvement
Can prioritize responses to client requests
I imagine such a protocol exists or could be implemented using WebSocket and storing state in a database.
Related
This is my issue:
I have an API which updates every 30 seconds the result displays on an endpoint
I want to display the result on my website for all visitors and so update it automatically every 30 seconds
I don't want each visitor to send a request on my API as it would overwhelm the API (and it is clearly not the right way to implement it I guess).
Is there a way to send one request every 30 seconds from my website backend to my API in order to display it for all visitors on the website?
Or maybe there is another smarter/efficient way to do it?
Another question I'm asking, I want exactly the same "front" website content for all users. I mean, there would have requests from frontend to backend to have the information, but I don't want some users to have the information earlier as their requests would be few seconds before other users. I was thinking to "send requests based on GMT+1 hour for example", I don't know if it makes sense or if there is another way ?
N.B.: I'm using Wix services, maybe I would have to change and build my proper website
Thanks a lot for your answers
Hugo
Is there a way to send one request every 30 seconds from my website backend to my API in order to display it for all visitors on the website?
You could use realtime communication between your website and your backend server. For example Socket.IO or SignalR depends on the stack you are using. So instead of sending a call every 30 seconds, the API server would dispatch an event to all clients and tell them to update (or send the data along when you dispatch the event).
I want exactly the same "front" website content for all users.
As for your second question, if you opt to use realtime communication to sync data between backend and frontend then there should be no need for having requests sent on a timer as your data is always updated actively from the server side.
I am developing a mobile application that will request Instagram server for searching photos with the hashtags.Client id is passed with the request.
Since the request limit provided by Instagram is 5000/hour so my question is:-
While requesting api of Instagram via client id rate limit is considered in respect of client id or IP address?
Please suggest me on the same.
I have read the doc but it was not mentioned there.
From the Instagram Developer site (endpoints):
Limits Be nice. If you're sending too many requests too quickly, we'll
send back a 503 error code (server unavailable).
You are limited to 5000 requests per hour per access_token or
client_id overall. Practically, this means you should (when possible)
authenticate users so that limits are well outside the reach of a
given user.
see also "Rate Limit Exceeded Instagram Error":
I tried to count my likes until i hit “Rate Limit Exceeded” Error, It
came around 100 – 150 Photos, and i waited for 5 mins and tried to
like again then the error goes off until i reach some 50 to 60 pics,
from the analysis i suppose Instagram is permitting to like 100 – 200
photos Per Hour.. and for Follows its 60 – 100…
I am getting this error whenever I try to follow someone on Instagram via API no matter how many follows have been done before:
{"meta":{"error_type":"APIError","code":400,"error_message":"Client request limit reached"}}
My app allows authenticated users to follow interesting people. I know that there is a 5000 call/hour limit per authenticated user, but it fails even with new users.
Do my app is reaching some kind of client level limit?
APIs like follow, unfollow, comment are limited to 350 requests per hour. However sending requests from client side will fix this problem to some extent but it allows the users to see your API token.
In this case it looks like it would be beneficial to get some more data from your users. You could use Google analytics to track the "follow" action
https://developers.google.com/analytics/devguides/collection/gajs/eventTrackerGuide
This would give you a timestamp and information about user behavior.
Even with an advertised rate of X requests per hour, one user hammering the service with your API key can cause everyone to get throttled. (Not guaranteed, but pretty common practice for companies to keep their services alive)
It might be a good idea to reset your API, its possible (though unlikely) that someone has acquired your key and is using it.
Assume there's a mobile app and a server.
I have question about rate limiting and hoping someone can give some advice on a design as I'm banging my head on how to navigate around rate limit. There must be something I"m missing because the 150 unauthenticated rate limit per IP per hour is extremely low.
Imagine the scenario I want to build is the following (simplified into a trivial example for this discusion). Assume user is signed into Twitter for this entire discussion to remove discussion about oAuth.
Mobile talks to our service to show users twitter friends list. Every time the mobile app is loaded, it will show the entire friends list, and highlighting the new friends that were added within the last 2 days.
That's it. But the trick is that I want to ensure that the friends list is always up to date in the client, which means our server has to have the most recent up to date friends list.
Periodically, I want my server to automatically scan the Twitter friends list for every user of my app to see if new friends have been added.
Our initial design was getting our server to do all the work with this flow:
New User signs in on client, gives access token to server
Server makes call to Twitter REST APIs to get initial friends lists
Server stores the Twitter Friends IDs and shows responds to the client with that list.
Periodically (e.g. every 48 hours), server checks Twitter REST APIs for friends list for each user and compares it to our cached Twitter friends list we have for them to see who is new and to highlight in the mobile app.
The good thing about this is that all the interaction with twitter to get friends list, compare and peridiocally refresh is on the server. Mobile client just makes a single call to my server and gets friends list.
The problem with this design is that it will work for a single user, but since the rate limit is 150 per hour on un-authenticated calls, I will hit my limit as soon as 151 users user my service (which has a fixed IP).
The only solution I can see is to have the client do the work for each user, then send me the friends list which my server caches. This takes care of Step #2 above. However, for Step #4, I'd have to build something into the client to auto refresh twitter friends and send back to the server.
This is super clumsy to have the client involved at all in this Twitter friends list operation.
At first I thought I was crazy and the public unauthenticated APIs like getting friends lists wouldn't be subject to rate limiting. However, according to their docs, it is.
Am I missing something obvious or is the only way to solve this is to put heavy logic into the client?
With whitelisting gone for those that aren't grandfathered or Twitter business partners, I don't think you have any alternative but to have your mobile app do the Twitter API calls from the handset.
Having the handset call Twitter isn't a bad thing by any means. Pretty much every Twitter client in the world does it. One benefit will be that the user will be authenticated to Twitter, and thus her full 350 calls per hour will be available to you. Keep in mind, however, that you should minimize your calls since the user may have other Twitter-aware applications installed on her handset eating into your call allotment, and vice versa.
Now to the solution. The way I would implement your use case would be to first fetch the complete list of friends for your user by calling the friends/ids method.
http://api.twitter.com/1/friends/ids.json?screen_name=yourUsersName
The above call will return the most recent 5,000 friend IDs, in order followed, for #yourUsersName. If you want to fetch more friend IDs than the first 5,000, you'll need to specify the cursor parameter to initiate paging.
Next, I would check the latest list of friends we just fetched against the list on the handset, syncing them by removing any IDs that are no longer present, while adding any that are new.
If we only need the friend IDs, then we're done at a cost of one API call per 5,000 friend IDs. If, however, we need to get user info for these new friends as well, then I would call users/lookup and pass in the list of all new users that we discovered while syncing friend IDs. You can request up to 100 user objects at a time.
http://api.twitter.com/1/users/lookup.json?user_id=123123,5235235,456243,4534563
You user must be authenticated in order to make the above request, but the call can fetch any Twitter user profiles you wish -- not just those that are friends of the authenticated user.
So, let's say for example that a user has 2,500 friends and has never used your app before. In that case, she would burn one call to fetch all of the friend IDs, and 25 calls for her friends' information. That's not too bad to get the app populated with data.
Subsequent calls should be more streamlined with probably only two calls burned (one for the IDs, and one to get the new friends).
Finally, once the data has been updated on the handset, the deltas for the IDs and user data can be gathered up and pushed to your server.
It may even be possible that your server application won't even have to interface with Twitter at all, and that should alleviate the 150 user limit you are encountering.
Some final notes:
Be sure to note in your app's privacy policy that you sync your user's friend list with your server.
I recommend specifying JSON as the return format for all Twitter API calls. It is a much more lightweight document format than XML, and you will typically transfer only about 1/3 to 1/2 as much data over the wire.
Pick a Twitter framework appropriate for your mobile device and your programming language. Twitter access is a commodity these days, and there's little to no reason to reinvent how to access the Twitter API.
I answered a similar question about an approach for efficiently fetching followers here.
Since you are making request on behalf of users you should make those requests be authenticated as those users. Then requests will count against each users own pool of 350 requests/hour.
I have heard that Twitter limits the number of API calls a third party app can make per hour. I believe the limit is around 100. My question is, does that limit apply per user, or is it 100 calls per app? 100 for the entire application seems very low, but I wanted to make sure and I couldn't find my answer in the documentation I was reading. Thanks.
https://dev.twitter.com/docs/rate-limiting
The default rate limit for calls to
the REST API is 150 requests per hour.
The REST API does account- and
IP-based rate limiting. Authenticated
API calls are charged to the
authenticating user's limit while
unauthenticated API calls are deducted
from the calling IP address'
allotment.
The limit as of Oct. 8, 2009 is 150 requests per hour per ip/account. Sending tweets doesn't go against that limit.
However, you can get your ip(s) and account whitelisted, which gets you up to 20,000 requests per hour per ip/account. To do that, go to: http://twitter.com/help/request_whitelisting
Note: You MUST be in production to be whitelisted.
For those of us in beta (myself included) all is not lost as Twitter has an API to check how many requests/hour you have left for that ip/account (the api is called rate_limit_status).
(Sorry, I'd post more link but Stackoverflow won't let me...)
http://apiwiki.twitter.com/Rate-limiting contains lots of information, including this;
"The default rate limit for calls to the REST API is 150 requests per hour. The REST API does account- and IP-based rate limiting. Authenticated API calls are charged to the authenticating user's limit while unauthenticated API calls are deducted from the calling IP address' allotment."
The rates are significantly lower than what they used to be. There is no more white listing, and most API calls are now 15 per 15 minutes. In addition there is a cap of calls per hour at 100, and an additional ceiling for things like direct messages; they are also limited to 1000 per 24 hours.
It is 100 calls per IP or per user. and the Limit was recently increased to either 125 or 150.
If you have a Twitter Application that comes from a single IP, like a web application, you can get your IP/Account Whitelisted, allow for 10,000 calls per hour.
It's 100 calls per user, not per application.
link text
It's most likely 100 calls per IP per hour. I doubt there's a reliable way for them to track which application is making the request.