We are planning to use Google Custom Search API to achieve one of our requirements. During the PT cycle we run into an issue of exceeding 100requests/100seconds/user limit (Since the API calls are happening on the Server Side,most likely the same limit is going to be exceeded in production as well). Is there a way to go around it(other than proxying API calls)? Ideally would be to white list the IP address so that the limit does not apply to it.
Will greatly appreciate any suggestion.
Roman.
Related
How can I get maps for addresses without requests limits ? Google provide only 2500 requests per day. First of all, I want to use free services. Thank you.
You left a ton of info out... What the heck is maps for addresses? Do you mean map tiles? Or are you talking about geocoding? Like getting addresses for maps.
Is it a website making the calls or mobile? Where are you exicuting the code from?
If you are talking about gps geocoding (getting an adress from a GPS cord) then there are tricks you can use to get around those limits. If it's based on a key then its a 2500 limit for the key. However, there are apis you can use that are based on calling IP (google is one) If you make the client make the call then unless your client is making 2500 calls your good to go.
You will notice here that the geocoding call doesn't require an api key. So the usagelimit is going to be based on calling IP
https://developers.google.com/maps/documentation/geocoding/#GeocodingRequests
Here's a similar question Usage limit on Bing geocoding vs Google geocoding?.
Google will start denying your request at around 2500. Bing has a much higher daily limit (used to be 30k - i think it's up to 50k now).
There are a number of free geo-coding services. I recommend staggering your requests to use multiple services if you need a large number of addresses coded daily. Here's a list of 54 providers: http://www.programmableweb.com/apitag/geocoding.
Have searched for answers on this for 2 days now with very little luck.
I'm developing a Drupal 7 site which has a Geofield field being autopopulated from an address field using the Google Geocoder API, but as of a couple of days ago this stopped working:
Exception: Google API returned bad status.\nStatus: OVER_QUERY_LIMIT in geocoder_google() (line 52 of /home/.../modules/geocoder/plugins/geocoder_handler/google.inc).
I can remove the proximity search filter that is sending too many requests to the Google API but I can't progress because I run into the above error every time I try to add a new record to the database (which just does one lookup to get a geocode from an address field but fails). Is there any way to unblock my site from Google's API or reset my usage? I've added an API key but to no avail. This was all working fine up until very recently, which I guess is when I unknowingly exceeded the usage limit.
I have limited API experience and am a Drupal/PHP beginner so please be gentle! Happy to provide more info, code, error messages etc if needed. Relevant Drupal 7 modules being used are OpenLayers, OpenLayers Proximity, Geofield, GeoPHP and Geocoder. Thanks for any help anyone can offer.
From Google Geocode Documentation:
Use of the Google Geocoding API is subject to a query limit of 2,500 geolocation requests per day. (User of Google Maps API for Business may perform up to 100,000 requests per day.) This limit is enforced to prevent abuse and/or repurposing of the Geocoding API, and this limit may be changed in the future without notice. Additionally, we enforce a request rate limit to prevent abuse of the service. If you exceed the 24-hour limit or otherwise abuse the service, the Geocoding API may stop working for you temporarily. If you continue to exceed this limit, your access to the Geocoding API may be blocked.
So, I guess you have to wait 24 hours, or upgrade to the business version.
Here's my issue. I have a site that provides some investing services, I pay for end of day data which is all I really need for my service but I feel its a bit odd when people check in during the day and it only displays yesterdays closing price. End of day is fine for my analytics but I want to display delayed quotes on my site.
According to the yahoo's YQL faq: If you use IP based authentication then you are limited to 1000 calls/day/IP, if my site grows I may exceed that but I was thinking of trying to push this request to the people browsing my site themselves since its extremely unlikely that the same IP will visit my site 1,000 times a day(my site itself has no use for this info). I would call a url from their browser, then parse the results so I can allow them to view it in the format of the sites template.
I'm new to web development so I'm wondering is it a common practice or a bad idea to have the users browser make the api call themselves?
It is not a bad idea at all:
You stretch up limitations this way;
Your server will respond faster (since it does not have to contact the api);
Your page will load faster because the initial response is smaller;
You can load the remaining data from the api in async manner while your UI is already responsive.
Generally speaking it is a great idea to talk with api's from the client. It's more dynamic, you spread traffic, more responsiveness etc...
The biggest downside I can think of is depending on the availability of other services. On the other hand your server(s) will be stressed less because of spreading the traffic.
Hope this helped a bit! Cheers!
I have 2 POST, 1 PUT and 1 DELETE API in my application. My application is deployed on Heroku. I want to rate limit these APIs but only to prevent against a DOS attack or in case someone by mistake calls API in infinite loop. What's the ideal rate limit for this scenario. E.g. x per minute, y per hour. What are the idea numbers for x and y?
One way to do it is to drop in the 3scale API plugin (https://github.com/3scale/3scale_ws_api_for_ruby) and this enforces rate limits, does analytics etc. using external infrastructure.
That way you can rate limit individual users and have different quotas for each one (plus all the signup etc.).
It won't strictly prevent true DOS because even unauthed request will still reach you - but it'll stop them going down into your stack + cut off the people who are doing the damage first.
This really depends on your app.
1) It depends on how "heavy" your app is:
If each of your requests are processor-heavy and/or hits your database a lot, then you'll want to set a low limit, because each request is "expensive". If your app is fast and efficient, then you have more room to maneuver and you can get set a higher API limit.
2) It depends on the use cases of your app.
Does your app require a lot of API hits to be useable? How does an API limit affect the features your app provides?
3) It depends on how many processors you have.
Heroku allows you to scale your app by setting web dynos and worker dynos. The more you have, the better you'll be able offer high API limits.
In any case, if you want to prevent someone from DOS-ing you or calling your API in an infinite loop, this is the wrong approach to take because an API limit affects the good guys as well as the bad guys. A better way is to detect the bad behaviour and respond appropriately (i.e. deny the offending IP address for a limited time).
Does anyone know where I can find Google API Request Limits for their different services?
On simulating 500+ concurrent users it seems to fail silently fairly often (maybe 1 in 10 loads)
Any ideas?
The information is in their support resources. I am not aware of a central place, but it's all there. Searching the docs for "request limit" should usually do the trick.
The Geocoding API's limits for example can be found here.
Google Maps API Web Services and Google Static Maps API limits were cut effective a few days ago. Starting October 1st 2011 commercial web sites and apps using Google Maps API for free receive:
max of 2,500 calls/day, if modified using Styled Maps feature
max of 25,000 calls/day in total
Fusion tables are preferable to the Google Maps API alone, particularly with respect to rate limits:
Applications using the Google Fusion Tables API can send a maximum of
5 requests per second to the Google Fusion Tables server.
I think they removed the limit recently: can't even find a mention of it in documentation pages where I know for sure that it was mentioned and read about the limit removal somewhere this summer.
Even their new EULA states that their service is not limited but they remain free to limit it however they want at any moment.
500 concurrent users doesn't seem to be that much though, even if limitations where in place; are you sure it's Google what's failing?