We are using the Google Calendar API to keep a sync between our app and events in our users' google calendar.
We have started regularly getting rate limiting errors (403).
However our usage according the APIs and Services page of the google cloud console is well below the stated limits (10,000 queries per minute and 600 per user per minute). We are also using the batch API to send our requests so cannot implement exponential backoff
Anyone got any advice on avoiding these rate limiting errors?
Rate limiting errors with google are basically flood protection you are going to fast. Dont hold to much stock in what the status shows on the Google developer console the numbers in those graphs are guesstimates at best and they are not Realtime.
The main cause for rate limit is that when you send a request there is no way of know which server your request is going to be run on. There is also no way of knowing what other requests are being run on the same server. So your request may run faster or slower than you would expect sometimes which makes it hard to track down exactly what 10,000 queries per minute and 600 per user per minute actually is.
10000 requests run on an overloaded server may run in 2 minutes while on a server that is not being overloaded it could be run in 30 seconds meaning the next request you send will blow out the quota.
As there is really no way of avoiding it you you should just ensure that your application is capable of responding to it by sending the request again. I wrote an article a number of years ago about how i would track my requests locally in my application and then ensured that it kept things at the right speed flood buster
Really as long as your application responds by sending the request again you should be ok.
Related
I am developing an app locally and I just integrated the Youtube data api v3 to query videos.
Last night I received the 403 error that said my daily quota has been exceeded. If I look at the chart under quotas in the developer console, it says there was 10,000 requests yesterday. This is totally impossible as I am only using this locally.
Here is the quota chart
If I click on credentials in the left hand menu and select the API key page, it says only 309 requests for that API key in the last 30 days. That is the only API key I have activated. It can't be API theft, as it only says 309 requests for that key.
I am totally confused. What is happening here? Is there a way to see the IP address where these requests are originating from?
Those 2 stats are different from each other. 1 request could have a quota cost of one to over a hundred. That quota cost reflects to the queries per day stat. So it's not surprising for you to hit 10,000 with 300 requests.
To get around this issue, you may need to optimize your API request to retrieve only the resource that you need. If the default quota (10,000) isn't really enough, then you probably need to request for quota increase through the console or through this direct apply for higher quota link.
Complete info can be found in the youtube documentation
You can check how to calculate on link
The problem: Running into Google Sheets API read/write quota limits. Specifically, the read/write requests per 100 seconds and read/write requests per 100 seconds per user quotas.
Some background:
For the past few months I've been developing a web app for students and staff in our school district which uses a Google spreadsheet as the database. Each school in our district was assigned a different Google spreadsheet, and a service account was created to make read and write calls to these spreadsheets on behalf of the web app.
We started with one school of approximately 1000 students, but it has now expanded to two other schools with a total user load of around 4000. Due to the nature of a school day schedule, we started hitting our quota limit (per 100 sec & per 100 sec per user) since almost everyone uses the app at the same time.
I found the usage limits guide for the Google sheets API, and as per the instructions I created a billing account, and linked the associated service account project to it. I then went to the quotas section in the developers console and applied for a higher quota. This involved filling out a Google form which asked "How much quota do you need? Express in number of API queries per day." Again, queries per day is not the problem, rather it's the number of queries per 100 seconds and per user (service account). After a couple of weeks our limit was increased to 2500 read/write requests per 100 seconds and 500 read/write requests per 100 seconds per user. The billing account was not charged, and after a little searching, I realized this was a free increase. This bump in our quota limit helped, but it's still going to be an issue because our district wants to add more schools in the future.
Here's what I need to know:
1) [ESSENTIAL QUESTION] Does Google have an upper limit or maximum to the number of read/write requests a single service account/user/IP can make within the 100 second time frame, and if so what is it?
2) If it is possible to go beyond our current quota limit (2500/500), is there another way of requesting/applying for the increase. Once again we have a billing account established for the project and are willing to pay for the service.
I've been pulling (what's left of) my hair out trying to find definitive answers to my questions. This post came close to what I was looking for, and I even did some of the things the OP suggested, but I just need a direct answer to my "essential" question.
Couple more things.
I understand that Google Charts Visualization doesn't have a quota limitation, and I'd consider using it however, for privacy reasons I can't have the spreadsheet keys exposed in plain javascript. Are there other options here?
Also, one might suggest creating multiple service accounts, but I'd rather avoid this if possible.
Thank you for your help. I'm very much a novice and I greatly appreciate your time and expertise.
To answer your questions:
1) [ESSENTIAL QUESTION] Does Google have an upper limit or maximum to the number of read/write requests a single service account/user/IP can make within the 100 second time frame, and if so what is it?
*The provided documentation only stated that Google Sheets API has a limit of 500 requests per 100 seconds per project, and 100 requests per 100 seconds per user. Check this post for additional information.*
2) If it is possible to go beyond our current quota limit (2500/500), is there another way of requesting/applying for the increase. Once again we have a billing account established for the project and are willing to pay for the service.
AFAIK, you can request for a higher quota limit and the Google Engineers may grant the request as long as you are making a reasonable request.
Also, you may check this thread for additional tips:
You can use spreadsheets.get to read the entire spreadsheet in a single call, rather than 1 call per request. Alternately, you
can use spreadsheets.values.batchGet to read multiple different
ranges in a single call, if all you need are the values.
The Drive API offers "push notifications", so you can get notified when changes occur and react to those, instead of polling for
them. The latency of the notifications is a little on the slow side,
but it gets the job done.
I was not able to find any information about the maximum number of concurrent requests and maximum number of requests in one second to the Visual Recognition service.
Can you provide me with information or link where I can read about limits in general?
There is not a hard limit on concurrency per user. The service is designed to support many users simultaneously, and therefore has capacity to process many requests in parallel. During periods of heavy use however you may occasionally receive a 500 return code, in which case the request should be resubmitted.
Unfortunately there is a legacy error message in the system that tells users they may be submitting too many concurrent requests , but it is very unlikely that the error is actually caused by a users' concurrency. They should be treated like a 500 error code - just resubmit the request.
You should have no problem submitting 30 or 40 requests concurrently.
Standard API keys are limited to 25,000 events per day by default to prevent something like an infinite loop from generating a huge bill. You can have that limit increased by creating a bluemix support ticket. If you really want to do some large scale processing in parallel it would also make sense to get in touch with Support with a bluemix ticket so that we can guide you in the most cost effective and efficient way to use the service.
I read that wikipedia's API is called MediaWiki. My question is regarding this API. Does this API have a maximum of calls per day/ hours / minutes ? I can't seem to find it.
See the wikimedia REST API "Terms and Conditions" for the latest rate limits (200 requests per second in 2022). What do you plan to do with the Wikipedia API?
They state some API:Etiquette and API:FAQ.
There is no hard and fast limit on read requests, but we ask that you
be considerate and try not to take a site down. Most sysadmins reserve
the right to unceremoniously block you if you do endanger the
stability of their site.
If you make your requests in series rather than in parallel (i.e. wait
for the one request to finish before sending a new request, such that
you're never making more than one request at the same time), then you
should definitely be fine. Also try to combine things into one request where you can (e.g. use multiple titles in a titles parameter instead of making a new request for each title
API FAQ states you can retrieve 50 pages per API request.
You can use Data Dumps as well if you need content offline (likely a little outdated).
For a graceful termination of your script in case you hit any of the limits, you can handle errors & warnings in API calls using these status messages.
If there is no need of a "live sample", it would be better to use a data-dump.
I have 2 POST, 1 PUT and 1 DELETE API in my application. My application is deployed on Heroku. I want to rate limit these APIs but only to prevent against a DOS attack or in case someone by mistake calls API in infinite loop. What's the ideal rate limit for this scenario. E.g. x per minute, y per hour. What are the idea numbers for x and y?
One way to do it is to drop in the 3scale API plugin (https://github.com/3scale/3scale_ws_api_for_ruby) and this enforces rate limits, does analytics etc. using external infrastructure.
That way you can rate limit individual users and have different quotas for each one (plus all the signup etc.).
It won't strictly prevent true DOS because even unauthed request will still reach you - but it'll stop them going down into your stack + cut off the people who are doing the damage first.
This really depends on your app.
1) It depends on how "heavy" your app is:
If each of your requests are processor-heavy and/or hits your database a lot, then you'll want to set a low limit, because each request is "expensive". If your app is fast and efficient, then you have more room to maneuver and you can get set a higher API limit.
2) It depends on the use cases of your app.
Does your app require a lot of API hits to be useable? How does an API limit affect the features your app provides?
3) It depends on how many processors you have.
Heroku allows you to scale your app by setting web dynos and worker dynos. The more you have, the better you'll be able offer high API limits.
In any case, if you want to prevent someone from DOS-ing you or calling your API in an infinite loop, this is the wrong approach to take because an API limit affects the good guys as well as the bad guys. A better way is to detect the bad behaviour and respond appropriately (i.e. deny the offending IP address for a limited time).